The Benchmark Pointed at Itself

We ended the last article on a promise: instrument the system, run the numbers, publish them either way. This is that follow-up. The numbers came back, and the real finding was not the number we expected to report.

The idea, in short: instead of letting an AI agent re-derive its approach from scratch on every run, promote the working code into a versioned registry and re-execute it. Stop paying for reinvention. Start paying once, and re-running the proven procedure. It's the same instinct behind packaging a Claude Code skill as a versioned, verifiable npm package instead of a copy-pasted folder: trust a specific version, not a vibe.

What we built

The registry is not a folder of scripts with good intentions attached. Every artifact carries a content digest computed over its own script bodies and its documentation, so an approval binds to exact bytes, not to a name. Every artifact carries a trust classification (verified or unreviewed, based on what the script touches, not what it claims to touch) and a health status (ok, stale, degraded, or unverified) that a run gate checks before anything executes. An unreviewed artifact refuses to run until a human signs off on its specific content digest. Change a single byte of the script, and that signature is invalid; the gate closes again automatically.

As of this run: 15 artifacts, 323 tests, 129 commits. Approvals are recorded as a signed, timestamped ledger entry per artifact. As of this week, that signature comes from a Secure Enclave key gated by Touch ID, specifically so an unattended process can't mint its own approval. That constraint isn't theoretical: earlier in the registry's life, an automated re-approval process minted signatures on content no human had reviewed, and the fix was to revoke every one of them and require a physical action to re-issue. It's the same governance instinct behind Rhize's skill governance work for Claude Code plugins: approve exact content, not a name that content can later stop matching.

Think of it the way a restaurant treats a recipe versus a chef's memory. A description of what the dish tasted like last time is not the recipe. The registry stores the recipe, not the tasting notes, and it refuses to serve a dish from a recipe nobody has read.

The first measurement

The paired run was against vault-inbox-processor, a real production workflow, on 2026-08-26. Arm A is the workflow with every step composed fresh by the LLM on every run, no registry involved. Arm B is the same workflow with its deterministic steps retrieved and re-executed from the registry instead of recomposed.

Two things in that table matter more than the headline number. First, 0.42 seconds isn't a rounding artifact: it's close to the entire deterministic surface of that workflow, measured directly rather than estimated. Second, the registry path was slower on that leaf, not faster. CLI startup, the digest check, the trust and health gates, and a Postgres round-trip all cost real wall-clock time, and at this scale that overhead dominates the tiny sliver of work being replaced.

In plain terms: we built a very careful, well-guarded way to skip a step that already took less than half a second. The guardrails cost more than the step did.

That's not the result we published this article to report. It's the honest one.

The meta-finding

Here is the part that earns the space in this piece. The benchmark's own row-append step (the step that writes the very row you're reading numbers from) was, until this run, an LLM-composed manual action, not a registry call. On 2026-08-23, a dedicated seeder run was supposed to append a row documenting an earlier scheduled run. It didn't. The note's row stayed empty. Nobody caught it at the time, because the failure produced no error, no red text, nothing. It simply produced less than it was supposed to.

Meanwhile, a verified, health-ok registry artifact that does exactly this job (append one row, refuse if the target section is missing, refuse if the row isn't pipe-delimited) had been sitting in the registry with zero production runs against it. The tool that would have caught the 2026-08-23 gap was built, approved, and idle.

A watchdog module built afterward to check for exactly this condition (row_missing: an expected row that never landed) caught it on its first live check.

The measurement system had the disease it was built to diagnose. We built a registry so that proven code replaces improvised recomposition, and the one place we forgot to apply that idea was the tool measuring whether the idea worked.

What the failures taught us

None of this was a one-off. Three verification lessons kept recurring across this project, in different disguises each time, and they generalize past this one registry.

An assertion that has never been proven able to fail is not a test. The fix is a neuter test, the same discipline behind Rhize's mutation-testing tooling for production code: take a script that's supposed to enforce something, mechanically strip out exactly the enforcement (nothing else), and confirm the check now fails where it used to pass. If the neutered version still passes, the check was never testing what you thought it was.

A control run has to prove its own mutation landed. A measurement that comes back clean and plausible looks identical, from the outside, whether the thing genuinely worked or the setup step that was supposed to change something never applied at all. One session on this project hit four separate broken probes in a row: a sed delimiter collision, a stray argument, misaligned patch indentation, an assertion that tripped on its own inserted text. Three of the four produced a confident, wrong conclusion before anyone caught it. The discipline that fixed it: between mutating something and measuring the result, confirm the mutation is present in the artifact. Not that the command exited zero. That the file differs.

Self-referential verification is the recurring disease, not an edge case. A critic that grades its own output, a smoke test that only exercises the version of the script currently in front of it, a benchmark that never checks whether its own row landed: these are all the same failure shape wearing different clothes. The fix is to anchor every check to a signal the thing being checked can't fake: an independent watchdog, a byte-level diff against a known-good artifact, a reviewer with no stake in the outcome.

Put simply: don't ask the thing you're grading to also hold the pencil.

Where determinism belongs

The 0.42-second number pointed the wrong direction at first read. It looks like it says there's almost nothing to gain here. What it says instead is that we went looking for determinism in the wrong layer.

A scheduled routine that calls an LLM step-by-step is already a graph. It has nodes (the steps) and edges (what runs after what, and under which condition). It's just drawn in prose inside a skill file, and an LLM re-reads that prose and re-decides the route on every single execution. Graph engineering is the act of compiling that graph instead of re-interpreting it: nodes get schema contracts for their inputs and outputs, the routes between them become code instead of paragraphs an interpreter has to re-parse, and every gate along the way has to be provably able to reject, the same neuter-test discipline from the section above, applied to routing decisions instead of scripts.

Under that framing, the leaves were never the problem. A single script re-run in under half a second is already about as deterministic as a step can get. The non-deterministic surface was always the structure around it: which step runs next, whether a gate blocks what it claims to block, whether the routing decision itself gets re-derived by an LLM every time or executed as compiled logic. That's where this registry is extending next, as a kind: graph artifact type alongside the existing skills and functions.

The code at the leaves was already deterministic. The map connecting the leaves was not, and the map was where the LLM kept redoing the same routing work, run after run.

This article is row three

This registry is one piece of a larger pattern in how we build with Claude Code, the same one behind the rest of Rhize's plugin work: don't just prompt better, build tooling that remembers what already worked.

The Procedural Memory Benchmark note had two rows in it before this article's run, both Arm A. This run appends the third, the first for Arm B. The deterministic stages of producing this specific article (the Sanity draft you're reading this in, the social drafts staged for review, and the row this paragraph describes) ran as retrieved registry scripts, not as LLM recomposition. The row recording that fact was appended by the artifact this article is about.

We said part two would publish the benchmark numbers whichever way they pointed. They pointed at the measurement system before they pointed anywhere else. That isn't a disappointing result. It's the result a real instrument is supposed to produce the first time you point it at something, including at itself.