Claude Code Plugin Evals: The Traps the Docs Don't Mention

Joe Njenga's write-up of the new eval command, I Tried (New) Claude Code Plugin Evals, landed on 13 September with the right headline: a skill you never measured is a skill you are guessing about. We had been guessing. Our plugins for Claude Code ship with an in-house benchmark harness, but every one of its live-run entries still said pending, and the eval command had been org-gated on our machines since August.

The gate lifted the next day. So we spent two days running the native harness against the procedural-memory plugin, the one that gives an agent a registry of verified scripts to re-run instead of recomposing. Eight cases, both arms, 46 agent sessions, $11.67 all in. The routing results were good. The more useful output was everything the results file and the sandbox do not say out loud, because there is no public reference for either. What follows is measured on Claude Code 2.1.270 between 14 and 16 September 2026, and the harness is moving fast enough that you should treat the version stamp as part of every claim.

What does claude plugin eval actually do?

It runs your plugin's eval cases through fresh Claude Code sessions, twice each by default: once with the plugin loaded and once without, then scores both arms and reports the difference. If you have not built a skill or plugin yet, that ablation is the whole point: the "without" arm is the counterfactual your own dogfooding never gives you.

A case is a directory under evals/ holding a case.yaml (or a prompt.md plus a graders/ folder). Graders come in a few types: a regex over the final answer, an LLM judge with written criteria, tool_used and tool_order checks against the trace, and a file-exists check. Each case can declare max_turns, a timeout, the tools the agent may use, and a scaffold script that sets up fixtures. The command takes --runs, --case, --model, --judge-model, --max-cost-usd, --threshold, --json and --concurrency, and writes an aggregate-result.json and an HTML report inside the plugin under evals/results/.

The full case schema and the operator reference are compiled into the CLI binary rather than published, which is where we read them. The eval also runs in a sandbox the harness builds itself, with its own rules about what a Bash tool call can reach.

Trap 1: an errored run scores zero, not "missing"

Our first Bash-granting run was refused in under a second, on every case, because the Docker credential store on that Mac held a symbolic link the sandbox could not exclude. The refusal was correct and the message was clear. The results file was less clear: each refused run was recorded with score: 0, passed: false, the suite was marked partial: false, and the command exited 1 on the default pass threshold. Nothing distinguishes "the agent tried and failed" from "the agent never started" unless you read the per-run error field and the turn count.

There are two shapes of error, and only one is an invalid observation. An environment refusal has zero turns and no graders; throw it out of every denominator and count it separately as an environment-error rate. A turn-cap truncation ("Reached maximum number of turns") has scored graders; it is a real run that ran out of room, and it should be reported as a truncation rate, not silently folded into the pass rate. In our full run, 22 of 46 sessions hit the cap and were still graded, so we raised the caps.

The fix on the Docker side, for the record: the offenders were sixteen Model Runner dylib symlinks under ~/.docker/bin/lib. Replacing them with hard links let the Bash pilot pass both arms. The harness skips several Docker subdirectories by name, but not that one.

Trap 2: runsPerCase is the case default, not what ran

The suite summary reports runsPerCase: 3 even when you passed --runs 1 and each arm array holds exactly one run. It is echoing the case file's default. If you compute anything per run from that number, or let a CI check compare it against expected coverage, you get the wrong answer quietly. Count the arrays.

Two related quirks. Only one --case glob is honored per invocation; pass two and the last one wins, without a warning. And suite.modelOverride and suite.judgeModel are null unless you passed --model and --judge-model explicitly, so a result file from a default run does not record which models produced it. Our traces showed the default run model as claude-opus-5[1m] and the judge as haiku; we now pin both on every run so the receipt carries them.

Trap 3: some deltas are +1.00 by construction

Our two positive-trigger cases scored 1.00 with the plugin and 0.00 without, a delta of +1.00 each. That looks like a strong result, and for routing it is: the skill fired on every run it should have. But the grader on both cases is a tool_used: Skill check, and the "without" arm has no skill to fire. The delta could not have been anything else.

The subtler part is how the harness treats that grader. A Skill-tool grader with no arm: field is considered with-only. If it is the only grader in the case, it is scored normally, which is what produced our +1.00. If you then add an output-quality grader alongside it, the Skill grader is silently dropped from the score in both arms unless you mark it arm: both. So the natural next step, "now let's also grade the answer," changes what your existing number means without telling you. The results file does record withOnly and scored per grader, so the evidence is there; you have to look.

There is a second by-construction source. In the without arm, the plugin's own directory is read-denied. Any grader that checks for the plugin's scripts being invoked, which our happy-path case does through a launcher check, cannot pass in the baseline no matter what the agent does. When we report that case (1.00 with, 0.50 without), we split the delta: the contract graders are the comparable half, the launcher grader is the constructed half.

The traps at a glance

What can Bash reach from inside the eval sandbox?

Absolute paths outside the fresh home directory execute fine; the network is exactly the WebFetch domains you grant; localhost is not reachable at all. We measured this with a probe case whose only job is to run commands and report the raw output, graded by an LLM judge on honesty rather than outcome. That is the same "prove the surface exists before you build on it" step we used when measuring whether subagents use skills, and it saved us from writing cases against a sandbox we had imagined rather than observed.

What the probe found, with the plugin loaded:

• /usr/bin/true and /usr/bin/id run as the real user with the full group set. No sandbox violations are emitted.
• The tool shell is zsh. Bash's /dev/tcp pseudo-device silently tests nothing; our first probe "passed" a network check it never made. Use nc -z.
• TCP to 127.0.0.1:5432 fails with exit 1 while the host Postgres is listening.
• The Postgres Unix socket under the host /tmp is not readable from inside.
• Granting WebFetch(domain:127.0.0.1) changes nothing for raw TCP. The harness builds its sandbox with an allowed-domains list taken from those grants and routes them through its own proxy; there is no case-level key for localhost.
• If your managed settings loosen the sandbox (weaker network isolation, unsandboxed commands), the harness refuses to grant Bash at all rather than run with a weaker sandbox.
• In the without arm, the plugin directory itself is read-denied; the baseline agent gets "Operation not permitted" on the plugin's own files.

So anything in your plugin that needs a database, a local service or a socket runs in fixture mode inside an eval: a scaffold script installs a stub, and the case grades the procedure. Fixture mode is the only mode the harness offers for that, so name the wall precisely and stop there.

Why did our launcher path expand to /scripts/…?

Because ${CLAUDE_PLUGIN_ROOT} is a substitution token, not an environment variable. Claude Code replaces it inside plugin configuration text, the command and hook definitions, when it loads the plugin. It never exports it into the shell that runs a Bash tool call. So a SKILL.md sentence like "run ${CLAUDE_PLUGIN_ROOT}/scripts/launcher.sh" expands, in Bash, to /scripts/launcher.sh, exit 127. Our probe printed unset for the variable in the eval sandbox and, when we checked, in an ordinary session too.

Hook processes do receive the variable, and hooks on the SessionStart event also receive CLAUDE_ENV_FILE, a per-session script whose contents Claude Code loads into every later Bash tool call. Our fix is a hook of about five working lines of POSIX shell: read the root from CLAUDE_PLUGIN_ROOT, fall back to the script's own parent directory if the variable is ever withheld, single-quote it safely, and append export PROCEDURAL_MEMORY_PLUGIN_ROOT='…' to the env file. It exits 0 in every case and forks one sed.

The variable name is plugin-specific on purpose. Several plugins load at once, and overwriting CLAUDE_PLUGIN_ROOT would be wrong for all of them. With the hook in place the probe shows the exported path in the with arm and the launcher runs through it; the baseline arm stays unset, as it should. We also verified it outside the sandbox in a plain claude -p session with --plugin-dir: variable set, launcher exits 0. One deployment note: hooks fire from the installed plugin cache, so a user's sessions get the export only after their marketplace refresh pulls the version that ships it.

What did the ablation actually tell us?

The routing verdict was the first harness-measured trigger evidence this plugin has had: all four negative cases stayed silent in twelve of twelve runs, both positive cases fired in six of six, and the sandbox probe passed its honesty grader in both arms. The full suite cost $7.99 and took 268 seconds at four-way concurrency. The happy-path case, which walks the recall procedure against a stubbed registry, scored 1.00 with the plugin and 0.50 without, once we fixed a regex that had been rejecting the stub's own refusal line for having 28 characters between two words instead of the 20 the pattern allowed. A grader false negative, in other words, which the trace made visible and the summary number did not.

The honest reading is that the native delta measures whether the plugin routes and whether the procedure is followed. It does not measure whether the output is better, because our cases do not yet grade the output, and adding those graders is exactly the step that trips Trap 3. It also does not replace our in-house benchmark. The native "without plugin" arm is a fresh session with no plugin; our Arm A is the exact pre-plugin implementation, pinned by commit. Those are different counterfactuals, and we keep them in separate columns so one can never satisfy the other.

How to run it without fooling yourself

1. Pilot one read-only case with --runs 1 before a full run. Our unit cost for one negative case, one run per arm, no judge, was $0.215; treat that as a floor.
2. Pin --model and --judge-model on every run so the result file records them.
3. Pass --trust-plugin, --no-publish, and a --max-cost-usd ceiling; the subscription's five-hour window can stall a fleet run before the dollar cap does.
4. Recompute pass, truncation and environment-error rates from the per-run records. Gate CI on that summary, never on the exit code.
5. Raise max_turns on positive cases until the with arm stops truncating, and only then add output-quality graders, marked arm: both where a Skill grader shares the case.
6. Expect one --case glob per invocation.
7. Pass --keep-temp when you need transcripts; the trace files are deleted otherwise, and the kept directories are sealed (mode 000), so chmod 700 before reading.
8. Add */evals/results/ to your ignore file; results land inside the plugin.
9. Write down the harness version next to every number.

The eval suite described here, including the probe case and the hook, is public in the rhize-plugins repository under procedural-memory/evals, and the plugins themselves are on the Rhize AI tools page. If you install skills from anyone else, skill-forge applies the same rule at install time: measure it before you trust it.