Give an AI Every Memory and It Does Worse

I thought it was a simple problem

ATLAS stores everything as nodes — a note, a retrospective, a decision, each is a node. In the previous post I built the structure that ties them together: types, tags, relations. And the AI reaches those nodes through MCP tools like load_memory and recall_knowledge — that memory lives behind a tool was a given from the start.

So the real question was what that tool should hand back. My first answer was simple: "Return all of it. The more it knows, the better."

Keyword search was too weak

Feeding everything was obviously too much, so the first thing I tried was keyword search — pull words from the question, find matching nodes.

The problem was that it barely worked. A note I had saved as "workout condition" wouldn't come up when I asked about my "body state." Same meaning, different words, and it slipped right past. A search that only finds the exact words you wrote is barely better than no search.

Keyword search was out.

The real problem was slowness

So maybe just return everything after all. I went back to that — and the issue wasn't wrong answers. It was that it was slow and expensive.

Even a trivial question sent the AI wading through hundreds of nodes. Simple asks took forever, and tokens poured out. The context window isn't infinite. Pushing hundreds of nodes and over a thousand relations in every time wasn't "knowing more" — it was re-opening the whole encyclopedia for every question.

What had to change was the inside, not the container

Keywords were too weak; everything was too heavy. The container — the MCP tool — stayed the same. What had to change was what it selected and returned, and in what order.

So the question flipped. Not "how do I fit it all in?" but "how do I decide what to leave out?"

How I designed the inside of the tool

First, I made it search by meaning instead of words. Every node is embedded into a vector ahead of time; the incoming question is embedded the same way; and the nodes closest in meaning (cosine similarity) are the ones that surface. Now "body state" finds the "workout condition" node — the meaning is close, even when the words aren't.

But close isn't the same as good. So on top of similarity I added judgment. Archived nodes are dropped (so an abandoned plan doesn't show up as current), tags have to overlap or the match is off-topic, and recent nodes are weighted higher. Two more things: a node that actually gets used stays "warm," while unused memory cools and sinks; and near-duplicate nodes get merged into one, which killed the "reading the same thing over and over" problem.

The tool's real job wasn't finding what's close. It was deciding what not to return.

What kinds of nodes ATLAS has

To follow the rest, you need the node types. ATLAS nodes come in two layers.

Memory nodes — where the content lives.

• Journal: raw entries, kept as written — the day, a conversation, a retrospective.
• Document: organized references — plans, decisions, longer writing.
• Insight: a distilled, reusable takeaway pulled out of those.

Meta nodes — where "how the AI should act" lives.

• Context: role and rules. agent-core ("you are this kind of assistant") and save-node-protocol ("record it this way") live here.
• Goal: what it is currently working toward.
• Lesson: something learned from being corrected, so the same mistake doesn't repeat.

Memory nodes are what recall_knowledge searches by meaning; meta nodes are what load_memory lays out at the start of a conversation.

Before retrieval, it needed a starting point

Even with good retrieval, the AI didn't know where to begin each conversation. So I made a rule: the first call, every time, is load_memory.

What it returns isn't one thing — it's a snapshot of "where you are right now." Who you are (Context), what you're working toward (Goals), what you've learned (Lessons) — the important nodes, laid out by priority. In the ranking, Context nodes sit highest, so agent-core always comes first.

From there, how to actually do something is checked by going into each node. When agent-core says "use this tool for images, load save-node-protocol before writing," the AI opens that node and reads the rule.

This is, in effect, metacognition — not knowing every record you hold, but first knowing what you know and how to act. What load_memory returns isn't data; it's the map the AI uses to understand itself.

If retrieval misses, the AI digs further

This approach has an obvious weakness: it only returns the top few, so if the node you needed isn't among them, it's missed.

But that is exactly why it's an MCP tool. If retrieval were a fixed prompt done once, a miss would be the end of it. When memory is a tool the AI calls, though, the AI searches again when it senses a gap — rephrasing the query, narrowing the tags, following neighboring nodes, opening the source. Retrieval stopped being something the system spoon-feeds once, and became an active loop the AI drives. That is the decisive difference between a static block of context and a tool.

In the end it was "what should the tool return"

So this post isn't about prompts. And it isn't about whether to use MCP — that was settled from the start. It's about how you design what the tool returns.

A good tool doesn't spill everything it holds. It does the selecting inside, hands back a focused result, and leaves the door open for the caller to dig deeper. Like a good assistant walking into a meeting who brings not your whole life but what today's agenda needs — and finds more when asked. Giving memory well turned out to be designing that tool well.

But reading wasn't enough — the AI started writing its own notes

Once retrieval was solid, something else surfaced. If load_memory is a mirror for "what I know," then who fills what the mirror reflects? Until now, I filled all of it — the Journals I wrote, the Documents I organized.

But for the assistant to do its own work, it had to record what it noticed too. The AI started writing its own Insight nodes, and turning corrections into Lesson nodes.

From memory to learning

This is where storage and learning split. Storage is piling up facts. Learning is different: when the AI gets corrected, it writes a Lesson — and in the next conversation that Lesson rides up the priority ranking and the AI actually behaves differently. The node doesn't just sit there; it changes the next action. Memory had started to update itself.

What came next

But all of this memory was text. The nodes, the retrieval, the learning — all of it happened over words.

My records aren't only words, though. Screenshots, diagrams, photos. For memory to really be memory, it had to remember images too.

Next post: teaching it to remember images.