The Benchmark Pointed at Itself

We ended the last article on a promise: instrument the system, run the numbers, publish them either way. This is that follow-up. The numbers came back, and the finding was not a number.

The idea, in short: instead of letting an AI agent re-derive its approach from scratch on every run, promote the working code into a versioned registry and re-execute it. Pay once for the invention, then re-run the proven procedure. It is the same instinct behind packaging a Claude Code skill as a versioned, verifiable npm package instead of a copy-pasted folder: trust a specific version, not a vibe.

What we built

The registry is not a folder of scripts with good intentions attached. Every artifact carries a content digest computed over its script bodies and its documentation, so an approval binds to exact bytes rather than to a name. Every artifact also carries a trust classification (verified or unreviewed, based on what the script touches, not what it claims to touch) and a health status (ok, stale, degraded or unverified) that a run gate checks before anything executes. An unreviewed artifact refuses to run until a human signs off on its specific digest. Change a single byte and that signature is invalid; the gate closes again on its own.

As of this run: 15 artifacts, 323 tests, 129 commits. Approvals are recorded as a signed, timestamped ledger entry per artifact, and the signature comes from a Secure Enclave key gated by Touch ID, so an unattended process cannot mint its own approval. That constraint is not theoretical. Earlier in the registry's life an automated re-approval process signed content no human had reviewed. The fix was to revoke every one of those signatures and require a physical action to re-issue them. It is the same governance instinct behind our skill governance work for Claude Code plugins: approve exact content, not a name that content can later stop matching.

In plain terms: the registry does not remember what an agent did. It locks a specific, hashed version of working code and refuses to treat any other version as approved until someone signs off on it by name.

The first measurement

The paired run was on 2026-08-26, across two production workflows. Arm A is the workflow with every step composed fresh by the model on every run, no registry involved. Arm B is the same workflow with its deterministic steps retrieved and re-executed from the registry instead of recomposed.

Two things in that table matter more than any headline number. First, 0.42 seconds is not a rounding artifact. It is close to the entire deterministic surface of the inbox workflow, measured directly rather than estimated. Almost everything else in that run is a model reading an inbox item and deciding what to do with it, so there is nearly nothing to freeze. Second, the Content Engine looks nothing like that. Roughly 55 percent of its wall time is deterministic work. The same mechanism meets a rounding error in one workflow and a large deterministic tail in the other, and the difference is the shape of the workload, not the registry.

On the inbox workflow the registry path was also slower on that leaf than the recomposed step it replaced. CLI startup, the digest check, the trust and health gates and a Postgres round-trip all cost real wall-clock time, and at the half-second scale the guardrails cost more than the step they guard. That is a fair observation about overhead at that scale. It is not a verdict on the registry, for the reasons in the next section.

Why this run cannot compare the arms

Two facts disqualify the run as an A-versus-B comparison, and we would rather say so than round it into a story about progress.

First, the model calls happened outside the graph runner. The timing and cost figures include model latency and provider behaviour that were not subject to the orchestration, retry logic or instrumentation the runner enforces on everything else. A number gathered under different conditions in each arm is not a comparison.

Second, the implementation changed during the benchmark drive. Code that was being measured was also being edited while the measurement was in progress. That is a snapshot of a moving target, not a controlled run.

So here is the line we are drawing. The deterministic share of each workload is a measurement we trust: it comes from the steps themselves, and it does not depend on which arm ran them. Any claim that Arm B was faster or slower overall, in either workload, does not survive those two conditions, and we are not making one.

The meta-finding

The benchmark's own row-append step, the step that writes the very row these numbers come from, was until this run an LLM-composed manual action rather than a registry call. On 2026-08-23 a dedicated seeder run was supposed to append a row documenting an earlier scheduled run. It did not. The row stayed empty. Nobody caught it at the time, because the failure produced no error and no red text. It simply produced less than it was supposed to.

Meanwhile, a verified, health-ok registry artifact that does exactly this job (append one row, refuse if the target section is missing, refuse if the row is not pipe-delimited) had been sitting in the registry with zero production runs. The tool that would have caught the gap was built, approved and idle.

A watchdog module built afterwards to check for exactly this condition, an expected row that never landed, caught it on its first live check.

The measurement system had the disease it was built to diagnose. We built a registry so that proven code replaces improvised recomposition, and the one place we forgot to apply that idea was the tool measuring whether the idea worked.

What the graph shakeout proved instead

The deliverable that did hold up was not a timing number. It was a shakeout of the graph itself: ten nodes completed end to end, and the pre-effect publish gate did its job. That gate sits in front of every node with a real-world side effect, and in this run it held back the Sanity writes, the HighLevel writes and the benchmark-recording effects until it explicitly cleared them. That is the runner's safety mechanism working under real conditions rather than in a unit test. How that gate fits the wider review workflow is covered in the piece on rhize-devflow's review gates.

What the failures taught us

None of this was a one-off. Three verification lessons kept recurring across this project, in different disguises each time, and they generalise past this one registry.

An assertion that has never been proven able to fail is not a test. The fix is a neuter test, the same discipline behind our mutation-testing tooling for production code: take a script that is supposed to enforce something, strip out exactly the enforcement and nothing else, and confirm the check now fails where it used to pass. If the neutered version still passes, the check was never testing what you thought it was.

A control run has to prove its own mutation landed. A measurement that comes back clean looks identical from the outside whether the thing worked or the setup step that was supposed to change something never applied. One session on this project hit four broken probes in a row: a sed delimiter collision, a stray argument, misaligned patch indentation, and an assertion that tripped on its own inserted text. Three of the four produced a confident, wrong conclusion before anyone caught it. Between mutating something and measuring the result, confirm the mutation is present in the artifact. Not that the command exited zero. That the file differs.

Self-referential verification is the recurring disease. A critic that grades its own output, a smoke test that only exercises the version of the script in front of it, a benchmark that never checks whether its own row landed: the same failure in different clothes. Anchor every check to a signal the thing being checked cannot fake, whether that is an independent watchdog, a byte-level diff against a known-good artifact, or a reviewer with no stake in the outcome.

Where determinism belongs

The 0.42-second number pointed the wrong direction at first read. It looks like it says there is almost nothing to gain. What it says is that we went looking for determinism in the wrong layer, and the Content Engine's 55 percent says the same thing from the other side: where the deterministic work is large, freezing it matters, and where it is tiny, the cost sits entirely in the steps freezing cannot touch.

A scheduled routine that calls a model step by step is already a graph. It has nodes (the steps) and edges (what runs after what, and under which condition). It is just drawn in prose inside a skill file, and a model re-reads that prose and re-decides the route on every execution. Graph engineering compiles that graph instead of re-interpreting it: nodes get schema contracts for their inputs and outputs, the routes between them become code instead of paragraphs, and every gate along the way has to be provably able to reject, the same neuter-test discipline applied to routing decisions instead of scripts.

Under that framing the leaves were never the problem. A single script re-run in under half a second is about as deterministic as a step can get. The non-deterministic surface was the structure around it: which step runs next, whether a gate blocks what it claims to block, whether the routing decision is re-derived by a model every time or executed as compiled logic. That is where the registry is extending next, as a graph artifact type alongside the existing skills and functions.

What the next cohort holds fixed

Before we run anything else, we are locking down the variables that made the first run uncomparable. For the next three trials, the following are held fixed:

• The graph itself, node for node
• The runner that executes it
• The model map: which model is assigned to which node
• Provider snapshots: the specific provider and version behind each model call
• Task scope: the exact work each trial is asked to do
• The input fingerprint: the exact input data each trial receives

If any one of these moves between trials we are back where this run left us, measuring something, but not the thing we set out to measure. It mirrors the logic of versioned, promoted tooling for agent-produced code, where a change to any dependency invalidates the promotion and forces a re-check.

We are not reporting or implying a performance gain from any of this. What is established is narrower: the deterministic share of two workloads, a proven publish gate, three verification rules, and a set of fixed conditions under which the next comparison can be trusted.

This article is row three

This registry is one piece of a larger pattern in how we build with Claude Code, the same one behind the rest of our plugin work: do not just prompt better, build tooling that remembers what already worked.

The Procedural Memory Benchmark note had two rows before this article's run, both Arm A. This run appended the third, the first for Arm B. The deterministic stages of producing this article (the Sanity draft you are reading, the social drafts staged for review, and the row this paragraph describes) ran as retrieved registry scripts, and the row recording that fact was appended by the artifact this article is about.

We said part two would publish the numbers whichever way they pointed. They pointed at the measurement system before they pointed anywhere else. That is the result an instrument is supposed to produce the first time you point it at something, including at itself.