mager.co

skill-evals is a small Claude Code plugin for evaluating the skills and agents you build. It came out of reading the awesome-evals PATTERNS playbook and wanting the patterns as something I could run inside a project, not just a reference to nod along to. v0.1 is two skills.

The first, error-analysis, is the unglamorous step most people skip: read 20–100 real traces, write a one-line note on the first thing that broke in each, cluster those into a handful of named failure modes, and rank them by frequency × severity. The point is that you can't write a good eval for a failure you haven't seen yet, and generic metrics like "helpfulness" point nowhere. The output is a short taxonomy that tells you what to actually measure — and which failures need a cheap code assertion versus an LLM judge.

The second, build-judge, is the one I learned the most from. Using an LLM to grade subjective things (tone, faithfulness, did-it-follow-the-instruction) is easy; trusting that grader is the hard part. On an imbalanced set — say 90% of outputs are fine — a judge that stamps everything "pass" scores 90% accuracy and catches none of the real failures. So the skill ships a stdlib-only score.py that reports true-positive and true-negative rate separately and gates on both, exiting non-zero below threshold so it drops straight into CI. The rubber-stamp judge fails that gate even at 90% accuracy, which is exactly the trap it's there to catch.

Since then, v0.2 shipped exactly the patterns I'd left out: add-assertions (deterministic check scaffolding), passk (pass@k versus pass^k with the unbiased estimator), synth-data (grounded synthetic sets when you have no traffic), and an eval-runner agent that runs a whole suite and reports honestly. I also did the dogfood — pointing eval-runner at all 124 posts on this very blog. The first pass screamed that zero posts passed every check, which is exactly the kind of number that makes you panic and learn nothing. The honest finding was that my spec was wrong, not the blog: it was grading against categories and a keyword format the real schema never used. Once I fixed the spec, every post came back clean. That's the lesson the whole thing is built to teach — most "failures" are your metric lying, and you only catch it by looking.

If you want the conceptual map behind these patterns rather than the code, I wrote one up: a plain-English tour of the eval types worth knowing.

Loooom is v1.0 — tagged and released on GitHub. The shape hasn't changed since the pivot post: fifteen curated non-technical skills, each scored against a written rubric. What v1.0 adds is the layer of testing that was missing.

The two rubric gates judge the skill text. The new third gate runs each skill the way an agent actually would — SKILL.md as the system prompt — and unit-tests its Agent Behavior contract with promptfoo: hook has to make you say what the song is about in one sentence before writing anything, stack has to kill the 23% credit card before entertaining the crypto bet, focus has to send your phone to another room. Thirty tests, two per skill, deterministic assertions plus an LLM rubric, all on Groq's free tier.

The whole harness still costs $0 — judge and tests both run on Groq's free tier. The price shows up in a different currency: run the suite three times back-to-back and you blow through the tokens-per-minute cap, and everything crawls behind HTTP 429s. Promptfoo's cache makes that mostly painless (passing tests don't re-run), but a free eval stack rations your iteration speed instead of your wallet. For a project this size, that's the right trade.

The first run came back 27/30, and the failures were the educational part. Two were the tests' fault, not the skills': story and frame were correctly following their own "make them name the one thing first" contract while my rubrics demanded the whole lecture in turn one. The third was a token cap truncating train before it reached progressive overload. Behavioral tests don't just check the skills — they force you to decide what the skill is actually supposed to do on the first turn.

The audit also closed an embarrassing loop: voice had been shipping without a worked example — the one skill not practicing what it preached, and the spec gate had been flagging it since day one. Fixed in v1.0.

I ran this whole launch with Claude Code on Fable, Anthropic's new model — the skill audit, the test suite, the release, and this note. First project I've shipped with it.

I run a Claude Code agent on a Mac mini in Chicago that I reach over Telegram. The hard part isn't the agent, it's keeping it up without me — across crashes, model swaps, and the occasional reboot. The fix is layered supervision, where each layer owns one kind of failure:

• run.sh loops the agent and watches its exit code. An in-session model switch exits with code 42; the loop sees that and relaunches on the new model. Any other code stops the loop and hands control up.
• tmux holds the session. The CLI is interactive and wants a PTY, so it runs inside a detached tmux session rather than a bare background process.
• launchd is the floor. A LaunchAgent with RunAtLoad starts the tmux session at login (so it survives a reboot), and a StartInterval watchdog re-checks every couple of minutes and rebuilds the session if it's gone.

The thing I keep relearning: "restart it when it dies" is not one job. A reboot, a crash, and an intentional model swap are different failures, and each wants a different layer to catch it. Pile them all into one script and it's brittle; separate them and the whole thing just stays up.