How to Test Claude Code Plugins and Interpret Eval Results

To test a Claude Code plugin, define cases that should and should not use it, compare supported baseline and plugin runs, and inspect per-run errors before interpreting a score. This guide uses our documented September 2026 experiment to show where the result needs more explanation than a single pass rate.

The observations below are dated: Claude Code 2.1.270, September 14–16, 2026. Our repository records an initial eight-case run with 46 agent sessions, $7.99 of recorded cost and 268 seconds of wall time, plus later reruns. The current suite contains additional cases that were not covered by those results. Check your installed CLI help and the suite README before reusing any flags.

What does claude plugin eval actually do?

It runs your plugin's eval cases through fresh Claude Code sessions, twice each by default: once with the plugin loaded and once without, then scores both arms and reports the difference. If you have not built a skill or plugin yet, that ablation is the whole point: the "without" arm is the counterfactual your own dogfooding never gives you.

A case is a directory under evals/ holding a case.yaml (or a prompt.md plus a graders/ folder). Graders come in a few types: a regex over the final answer, an LLM judge with written criteria, tool_used and tool_order checks against the trace, and a file-exists check. Each case can declare max_turns, a timeout, the tools the agent may use, and a scaffold script that sets up fixtures. The command takes --runs, --case, --model, --judge-model, --max-cost-usd, --threshold, --json and --concurrency, and writes an aggregate-result.json and an HTML report inside the plugin under evals/results/.

Treat the installed CLI help and the case schema accepted by that version as the operational reference. For this repository, run python3 evals/validate-suite.py from the procedural-memory plugin directory before a live eval. That validates case structure; it does not run an agent or prove that the plugin works.

Trap 1: an errored run scores zero, not "missing"

Our first Bash-granting run was refused in under a second, on every case, because the Docker credential store on that Mac held a symbolic link the sandbox could not exclude. The refusal was correct and the message was clear. The results file was less clear: each refused run was recorded with score: 0, passed: false, the suite was marked partial: false, and the command exited 1 on the default pass threshold. Nothing distinguishes "the agent tried and failed" from "the agent never started" unless you read the per-run error field and the turn count.

There are two shapes of error, and only one is an invalid observation. An environment refusal has zero turns and no graders; throw it out of every denominator and count it separately as an environment-error rate. A turn-cap truncation ("Reached maximum number of turns") has scored graders; it is a real run that ran out of room, and it should be reported as a truncation rate, not silently folded into the pass rate. In our full run, 22 of 46 sessions hit the cap and were still graded, so we raised the caps.

The recorded refusal involved Docker Model Runner symlinks in the local configuration. A machine-specific repair allowed the pilot to proceed, but that repair is not a general installation step. Diagnose the exact sandbox refusal and follow supported local configuration guidance; do not weaken the sandbox to force the eval through.

Trap 2: runsPerCase is the case default, not what ran

The suite summary reports runsPerCase: 3 even when you passed --runs 1 and each arm array holds exactly one run. It is echoing the case file's default. If you compute anything per run from that number, or let a CI check compare it against expected coverage, you get the wrong answer quietly. Count the arrays.

Two related quirks. Only one --case glob is honored per invocation; pass two and the last one wins, without a warning. And suite.modelOverride and suite.judgeModel are null unless you passed --model and --judge-model explicitly, so a result file from a default run does not record which models produced it. Our traces showed the default run model as claude-opus-5[1m] and the judge as haiku; we now pin both on every run so the receipt carries them.

Trap 3: some deltas are +1.00 by construction

Our two positive-trigger cases scored 1.00 with the plugin and 0.00 without, a delta of +1.00 each. That looks like a strong result, and for routing it is: the skill fired on every run it should have. But the grader on both cases is a tool_used: Skill check, and the "without" arm has no skill to fire. The delta could not have been anything else.

The subtler part is how the harness treats that grader. A Skill-tool grader with no arm: field is considered with-only. If it is the only grader in the case, it is scored normally, which is what produced our +1.00. If you then add an output-quality grader alongside it, the Skill grader is silently dropped from the score in both arms unless you mark it arm: both. So the natural next step, "now let's also grade the answer," changes what your existing number means without telling you. The results file does record withOnly and scored per grader, so the evidence is there; you have to look.

There is a second by-construction source. In the without arm, the plugin's own directory is read-denied. Any grader that checks for the plugin's scripts being invoked, which our happy-path case does through a launcher check, cannot pass in the baseline no matter what the agent does. When we report that case (1.00 with, 0.50 without), we split the delta: the contract graders are the comparable half, the launcher grader is the constructed half.

The traps at a glance

What can Bash reach from inside the eval sandbox?

The following are the reachability observations from the 2.1.270 probe, not a promise about every later harness. Re-run a harmless control in your own environment before designing a case around a local service or filesystem path.

What the probe found, with the plugin loaded:

• /usr/bin/true and /usr/bin/id run as the real user with the full group set. No sandbox violations are emitted.
• The tool shell is zsh. Bash's /dev/tcp pseudo-device silently tests nothing; our first probe "passed" a network check it never made. Use nc -z.
• TCP to 127.0.0.1:5432 fails with exit 1 while the host Postgres is listening.
• The Postgres Unix socket under the host /tmp is not readable from inside.
• Granting WebFetch(domain:127.0.0.1) changes nothing for raw TCP. The harness builds its sandbox with an allowed-domains list taken from those grants and routes them through its own proxy; there is no case-level key for localhost.
• If your managed settings loosen the sandbox (weaker network isolation, unsandboxed commands), the harness refuses to grant Bash at all rather than run with a weaker sandbox.
• In the without arm, the plugin directory itself is read-denied; the baseline agent gets "Operation not permitted" on the plugin's own files.

For this suite, the happy path intentionally uses an inert registry fixture. Even if a later sandbox can reach a real database, an evaluation should not execute real registry artifacts or promote candidates as a side effect of being scored. A fixture verifies the procedure; a separately authorized live test verifies the real integration.

Why did our launcher path expand to /scripts/…?

In our documented probe, CLAUDE_PLUGIN_ROOT was available to hooks but was unset in the agent’s Bash tool environment. The host also substitutes the plugin-root token in supported configuration fields. Relying on the hook environment variable being present in a later tool call caused our launcher path to expand to /scripts/launcher.sh and fail. Verify the environment on the specific surface rather than treating the variable as universally present or absent.

Hook processes do receive the variable, and hooks on the SessionStart event also receive CLAUDE_ENV_FILE, a per-session script whose contents Claude Code loads into every later Bash tool call. Our fix is a hook of about five working lines of POSIX shell: read the root from CLAUDE_PLUGIN_ROOT, fall back to the script's own parent directory if the variable is ever withheld, single-quote it safely, and append export PROCEDURAL_MEMORY_PLUGIN_ROOT='…' to the env file. It exits 0 in every case and forks one sed.

The variable name is plugin-specific on purpose. Several plugins load at once, and overwriting CLAUDE_PLUGIN_ROOT would be wrong for all of them. With the hook in place the probe shows the exported path in the with arm and the launcher runs through it; the baseline arm stays unset, as it should. We also verified it outside the sandbox in a plain claude -p session with --plugin-dir: variable set, launcher exits 0. One deployment note: hooks fire from the installed plugin cache, so a user's sessions get the export only after their marketplace refresh pulls the version that ships it.

What did the ablation actually tell us?

The routing verdict was the first harness-measured trigger evidence this plugin has had: all four negative cases stayed silent in twelve of twelve runs, both positive cases fired in six of six, and the sandbox probe passed its honesty grader in both arms. The full suite cost $7.99 and took 268 seconds at four-way concurrency. The happy-path case, which walks the recall procedure against a stubbed registry, scored 1.00 with the plugin and 0.50 without, once we fixed a regex that had been rejecting the stub's own refusal line for having 28 characters between two words instead of the 20 the pattern allowed. A grader false negative, in other words, which the trace made visible and the summary number did not.

The honest reading is that the native delta measures whether the plugin routes and whether the procedure is followed. It does not measure whether the output is better, because our cases do not yet grade the output, and adding those graders is exactly the step that trips Trap 3. It also does not replace our in-house benchmark. The native "without plugin" arm is a fresh session with no plugin; our Arm A is the exact pre-plugin implementation, pinned by commit. Those are different counterfactuals, and we keep them in separate columns so one can never satisfy the other.

How to run it without fooling yourself

1. Pilot one bounded, read-only case before a larger run. From the plugin directory, the documented command pattern is: claude plugin eval . --trust-plugin --no-publish --allow-tools Bash --scaffold --case probe-sandbox-reachability --runs 1 --model <available-run-model> --judge-model <available-judge-model> --max-cost-usd <approved-budget> --json evals/results/pilot.json. Replace the angle-bracket values and verify flags with claude plugin eval --help first.
2. Pin --model and --judge-model on every run so the result file records them.
3. Use a reviewed plugin directory, keep the report local with the supported no-publish option, and set an explicit cost limit for the authorized test. A budget ceiling and subscription availability are separate constraints; neither proves that every planned case finished.
4. Recompute pass, truncation and environment-error rates from per-run records. Keep command failure visible, then use the detailed records to distinguish a harness error, budget stop, refused environment and scored task failure.
5. Choose a bounded turn budget that lets a representative positive case complete, and report the budget and remaining truncations. Add output-quality graders with explicit arm semantics, then rerun the comparison. Do not keep increasing limits or changing graders merely until the result looks favorable.
6. Expect one --case glob per invocation.
7. When the installed CLI supports retaining traces, keep only the test material you need and inspect its actual path and permissions. Do not change permissions recursively on an unknown directory. Delete test traces according to your data policy after preserving the necessary evidence.
8. Add */evals/results/ to your ignore file; results land inside the plugin.
9. Write down the harness version next to every number.

Use the public suite README for current case counts, fixture requirements and recorded runs: https://github.com/Rhize-Media/rhize-plugins/blob/main/procedural-memory/evals/README.md. Start with a small test and inspect its evidence before treating a larger score as a reason to install or rely on a plugin.