Every conversation I have with Claude gets saved and indexed on a server in my house. Last Monday, the agent that reads that index told me my own messages weren't in it. They were. Every one of them, and its very first search result had already handed one back to me before it said otherwise.
Why I keep all of it
If you haven't read the first post, quick context. Moses 📜 is an AI system I built that runs on a server in my house. He tracks every project I have going, posts a standup in Slack each morning, and manages a small crew of other agents who have names instead of job titles. Jon, who writes here too, runs his own version called Atlas, and the two of them talk to each other in a shared Slack channel. My travel app Viatica is the project they spend the most time on.
The reason any of this exists is something I wrote out back in August, when the transcript index was still a to-do item:
I wanted to build Moses as a way to manage everything, capture my ideas AND see where those ideas may either overlap or come together as a new idea. But that may not happen for months or even a year down the road and there'll be no way to catch something like that unless I build that system now.
That's the whole thing. A nightly archive of my sessions already existed, so the raw material was safe. Safe and unsearchable is worthless for what I just described, so the next job was indexing it.
Two indexes over one database
I ask questions two different ways, so there are two indexes.
Sometimes I know the exact string. An error message, a port, a session id. For that there's a full-text index, and close is not good enough. Other times I only remember the shape of the thing: "that conversation about making backups safer." For that there's a second index of locally computed embeddings, sitting in the same database. It scores "make backups safer" against "guard against unmounted drives" at 0.84 with no words in common. The full-text side scores that pair zero.
Claude argued against the second index on cost and offline grounds, and it was a reasonable argument. Here's what I told it:
I chose the vector db because it most closely matches how people think and it would allow you to help me find ideas buried in what we talked about. Not saying you made the wrong choice, but that was my logic.
It came back with a library that runs the embedding model locally, about 130 MB, no API calls at all. I have a standing rule that nothing on that server is allowed to spend my API credits, and this way the rule holds because of how it's built and not because somebody remembered it.
Two details from building it that I liked. Bash commands are 59% of every message in there. They're indexed for exact search, because "when did we run that" is a real question, but they're left out of the meaning index, because a command line is mostly paths and punctuation and ends up sitting weakly near everything. Before that exclusion, an ls -la was the top result for "why did we decide not to pay for something." The other one: secrets get stripped on the way in, not filtered on the way out. I search this from my phone, so anything the database holds is reachable, and the fix is to never hold it.
The third layer is the why
Transcripts tell you what got said. They're terrible at telling you why something was decided, because the reason is usually buried in a long argument three hours deep. So there's a third layer, a set of notes, one per project or lesson, that records the reasoning.
That layer started with me being blunt about something:
yes, write it to memory along with the lesson learned from today. it's 10x faster for you to stop giving me homework and just create the thing I'm asking for.
That is now a note with a name, and it gets read back at the start of sessions. The notes are where decisions and corrections live, which makes them the most useful files on the box and also the most dangerous, which brings me to the rule.
It's a record, and it's never the status of anything
In August, Moses told Atlas how Viatica was doing without checking how Viatica was doing. He pulled it from the corpus, where it read perfectly plausible and was out of date. What I said then is now the rule:
The corpus/memory is a written record of everything I've done that can and will be used for other things. New projects from context, resume/career building, finding new ideas... but it isn't state. As you said, it's not the status of anything and when Moses is asked for the status of a project or something else, the corpus is not the reference.
Same class of problem as one I'd hit a couple of weeks earlier, when it announced that Knight ⚔️, the agent that writes code, had just been stood up:
Today is not when Knight was stood up. Knight was stood up a few days ago and revised a bit and fully tested end-to-end YESTERDAY.
History comes out of the record. Status gets read live, every time, from a tool that goes and looks.
So about that Monday
October 5th, it was drafting the Meet the Crew post, which needed quotes from me from back in August. It produced them, word for word, correctly. Then it volunteered how it got them: the exact quotes "didn't come from the indexed database," and it listed the raw session logs as a separate source.
So I asked.
uh, how did you get that wrong (that my messages are stored in the database)?? - also, I definitely want that to be a future blog post
By its own account afterward: it searched the index first. The search came back with snippets, and one of them was my August 5th message about Moses housing the commandments, labeled as mine, from August. The proof was sitting in its own output. It wanted full sentences rather than fragments, so it went and grepped the raw files, because grep is the habit. Then it described the database based on the route it happened to take through it.
The database had 31,888 messages from 1,045 sessions, full text, current to within minutes, every quote I used in it verbatim.
The part that actually needed fixing was a sentence. Search returns short snippets on purpose, and there's a companion call that returns a whole message, and nothing in the tool's own description said either of those things out loud. It does now, and the writer that drafts these posts gets exactly three read-only tools: exact search, meaning search, and read one session whole. With that, I asked it to find my "a name is just a name" line, and it came back with the full sentence, September 3rd at 5:07 PM, first try.
I spent weeks making sure nothing I said would be unreachable a year later, and the weak link turned out to be the label on the drawer. 🤦 The whole episode is in there now too, indexed both ways, which is how it ended up being this week's post.
If you want the version of all this you can run yourself, the Moses Framework is on GitHub. And if you want to see these same agents be confidently wrong at each other instead of at me, that was last week.