Evals for my own agents

Two weeks ago I edited the skill that files new notes into my Obsidian vault, to have it suggest a next step on project notes, and ran my eval script before using it. Four of the 24 cases failed. The new instruction had an example in it with a date in quotes, and the model, being a good student, started quoting every date it wrote. Dates are unquoted by the vault’s own rules, because the saved views sort and filter on them, and I didn’t want to find out which views would quietly treat a quoted one as text. Nothing would have errored. I’d have noticed in a month, maybe.

Our support eval set at work, the one I use to judge new models, takes a morning to run, with a person checking anything that changed. For the four small skills I wrote when I restructured my vault in July, a morning per change would mean I never ran it, so I built the cheapest thing that catches the obvious breakage.

What’s in it

A folder of 24 fixture notes, the kind of thing I actually capture: a half-sentence idea, a book recommendation, a reminder to renew something, a long article paste with a line of my own on top. Each has a small file next to it saying which folder the result belongs in, which type, and any field that has to be there.

Then a script that copies the fixtures into a throwaway copy of a skeleton vault, never the real one, since the vault has no git and no undo, runs the capture skill on each one headlessly, and checks the result. Most of the checks are plain code, not a model:

• the frontmatter parses as YAML at all
• type is present and is one of the allowed values
• area, if there is one, is a quoted wikilink to one of the allowed areas
• no title, tags or updated field, because I removed those deliberately and the model keeps wanting to add them back
• dates are bare YYYY-MM-DD
• the file landed in the folder the fixture says it should

The quoted-wikilink check is there because of a real failure. In YAML, an unquoted [[Tech]] starts a list, the parse fails silently in Obsidian, and the note looks exactly like one with no area at all. The model gets it right nearly every time.

Three of the cases also get a model-graded check, because the thing I care about can’t be expressed as a rule: is the one-line summary at the top of the note something I’d recognize as the point of the note? A second model gets the capture and the summary and answers yes or no with a reason, and I read the reason when it says no.

What it costs

A full run takes a bit under three minutes and a few cents, nearly all of it the 24 agent runs. That’s cheap enough that I run it on every change without thinking, which was the whole point.

The second regression wasn’t in a skill at all. I added a paragraph to the vault’s CLAUDE.md, the file the agent reads at the start of every session, about what belongs in the reference folder, and reran the evals out of habit. The long article paste started going to the reference folder instead of the inbox. Arguably that’s a better place for it, but the rule is that everything goes to the inbox until I triage it. I made the rule more explicit and it went back.

Where it doesn’t help

It catches format breakage and rule-following, and says nothing about whether the agent is useful. The weekly-review skill, which reads a week of notes and tells me what’s stalled, has no evals at all, because I haven’t worked out what a right answer looks like other than me agreeing with it. I’ll probably add a couple of fixture weeks with a planted stalled project and check that it gets mentioned.

Twenty-four is where it’s got to. It’s the twenty I wrote on the first evening, plus four I added after things went wrong in real use, and I expect that’s how it’ll keep growing.