AI Workflow Benchmarks: Why Our First Comparison Was Invalid

Our first AI workflow comparison could not support a faster-or-cheaper conclusion. Model calls ran outside the instrumented graph and the implementation changed during measurement. The useful result was identifying those weaknesses and recording the fixed conditions needed for another trial.

We ended the last article on a promise: instrument the system, run the numbers, publish them either way. This is that follow-up. The numbers came back, and the finding was not a number.

The proposed approach is to turn a repeatable part of an AI workflow into versioned code, then reuse it under explicit checks. This can reduce repeated composition, but it also introduces setup, lookup and validation overhead. Whether that tradeoff helps must be measured on the intended workload.

What we built

The historical design bound approval to a content digest covering scripts and documentation, then checked trust and health before execution. That is a useful design contract: changing the content should invalidate the old approval. The contract needs direct tests, including privileged and recovery paths; documentation alone does not establish that it held in a run.

The original project account described a failure in which automated approval did not correspond to human review, followed by a move to an approval path requiring user presence. The lesson is to distinguish a cryptographic signature from evidence that someone reviewed the exact content. This article does not re-verify those historical approval events.

In plain terms: the registry does not remember what an agent did. It locks a specific, hashed version of working code and refuses to treat any other version as approved until someone signs off on it by name.

The first measurement

The historical comparison contrasted freshly composed workflow steps with deterministic steps retrieved from a registry. The intended question was whether reuse changed the work required for an equivalent outcome. The underlying receipts were not available in this review, so numerical timings and workload shares have been removed.

Before measuring a benefit, classify the work. A workflow dominated by interpretation has a different opportunity from one with substantial repeatable formatting, validation or storage. Do not estimate the deterministic share from the label of the workflow; instrument the steps and keep the trace.

Reuse has overhead: process startup, content verification, approval checks and storage access. Those costs may be material for a short operation. Measure the end-to-end path rather than assuming a reused script is faster than a newly composed step.

Why this run cannot compare the arms

Two facts disqualify the run as an A-versus-B comparison, and we would rather say so than round it into a story about progress.

First, the model calls happened outside the graph runner. The timing and cost figures include model latency and provider behaviour that were not subject to the orchestration, retry logic or instrumentation the runner enforces on everything else. A number gathered under different conditions in each arm is not a comparison.

Second, the implementation changed during the benchmark drive. Code that was being measured was also being edited while the measurement was in progress. That is a snapshot of a moving target, not a controlled run.

The invalid comparison does not support a faster-or-slower conclusion. Without the underlying records, this review also cannot certify the earlier deterministic workload shares. We retain the design problem and the conditions needed for a repeatable comparison, not the numerical claims.

The meta-finding

The original account also described a missing benchmark-recording step: work was expected to append a result row, but the row was absent. That is an important failure class even when a process returns no obvious error. Define completion in terms of the saved artifact, not just the final assistant message.

A purpose-built row writer can reject a missing target section or malformed row, but merely having the tool available does not prove a workflow called it. The runner needs an explicit invocation and a check of the persisted result.

An independent completion check should compare expected effects with observed state. It should distinguish a missing result from a delayed or ambiguous write before proposing a retry.

The measurement system had the disease it was built to diagnose. We built a registry so that proven code replaces improvised recomposition, and the one place we forgot to apply that idea was the tool measuring whether the idea worked.

What a graph shakeout should verify

A graph shakeout should exercise transitions and permission gates before it is used for a performance claim. Place approval checks before external writes, test a deliberate refusal, and verify that the destination stayed unchanged. Source tests and a successful demonstration need separate evidence; this review does not certify the historical publish-gate run.

What the failures taught us

None of this was a one-off. Three verification lessons kept recurring across this project, in different disguises each time, and they generalise past this one registry.

An assertion that has never been proven able to fail is not a test. The fix is a neuter test, the same discipline behind our mutation-testing tooling for production code: take a script that is supposed to enforce something, strip out exactly the enforcement and nothing else, and confirm the check now fails where it used to pass. If the neutered version still passes, the check was never testing what you thought it was.

A control experiment must prove its own setup change landed. A patching command can fail because of quoting, indentation or a mismatched marker while later checks still run. Inspect the resulting artifact and compare it with the intended control before interpreting the outcome. Exit status alone may not establish the mutation you meant to test.

Self-referential verification is the recurring disease. A critic that grades its own output, a smoke test that only exercises the version of the script in front of it, a benchmark that never checks whether its own row landed: the same failure in different clothes. Anchor every check to a signal the thing being checked cannot fake, whether that is an independent watchdog, a byte-level diff against a known-good artifact, or a reviewer with no stake in the outcome.

Where determinism belongs

The practical question is where repeatable work sits in the process. Freezing a small leaf operation cannot address time spent deciding which step comes next. Measure both the leaf and the orchestration path before choosing what to compile into code.

A scheduled routine that calls a model step by step is already a graph. It has nodes (the steps) and edges (what runs after what, and under which condition). It is just drawn in prose inside a skill file, and a model re-reads that prose and re-decides the route on every execution. Graph engineering compiles that graph instead of re-interpreting it: nodes get schema contracts for their inputs and outputs, the routes between them become code instead of paragraphs, and every gate along the way has to be provably able to reject, the same neuter-test discipline applied to routing decisions instead of scripts.

Routing can be made explicit through schemas, transitions and gates instead of being reinterpreted from prose on every run. That reduces one source of variability, but the graph still needs correct inputs, failure handling and tests. It does not make model-generated content deterministic.

What the next cohort holds fixed

For a valid next comparison, freeze the following before collecting the main cohort:

• The graph itself, node for node
• The runner that executes it
• The model map: which model is assigned to which node
• Provider snapshots: the specific provider and version behind each model call
• Task scope: the exact work each trial is asked to do
• The input fingerprint: the exact input data each trial receives

If any one of these moves between trials we are back where this run left us, measuring something, but not the thing we set out to measure. It mirrors the logic of versioned, promoted tooling for agent-produced code, where a change to any dependency invalidates the promotion and forces a re-check.

The useful output of this account is a measurement checklist and a record of why the initial comparison was not valid. It does not establish a deterministic workload share, a verified publish gate or a performance advantage. Revisit those questions only with retained, independently reviewable run evidence.

Make the measurement record part of the workflow

This registry is one piece of a larger pattern in how we build with Claude Code, the same one behind the rest of our plugin work: do not just prompt better, build tooling that remembers what already worked.

A benchmark record should identify the protocol, input fingerprint, implementation revision, attempt and outcome. Verify that the record was written exactly once and that an ambiguous write is reconciled before retrying. The record should remain inspectable after the assistant session ends.

An early evaluation can be useful when it reveals that the measurement system is incomplete. Keep that correction visible, repair the method and collect a new frozen cohort. Do not turn a failed comparison into an implied performance win.