Managing AI Agent Skills: A Case Study in Preventing Duplication

Managing a large AI skill library means knowing which skills overlap, which copies have drifted and which tool owns each job. A generated relationship map can make those questions inspectable. This case study explains how we applied that approach after finding a stale fork in our own library, and what another team can test before adopting it.

The skill was a fork of an open-source original called strategic-compact, pulled into our Claude Code plugin marketplace because we wanted to tune it. After we forked it, the upstream project kept moving: it gained a context-size signal with thresholds that scale to the model's context window. Our copy still ran the old logic, which only counted tool calls. Anyone invoking our version was getting a strictly worse skill than the one sitting in the upstream repo, free, one install away.

Here is the part that actually stung. When we audited the other skills we had forked from the same source, three of them differed from their upstreams only in frontmatter indentation. We had taken on the full maintenance burden of four forks and three of them added literally nothing. A fork's staleness is invisible until someone diffs it, and nobody diffs anything until something breaks.

That incident became a rule in our repo's CLAUDE.md: close gaps, never duplicate. But a rule written in prose is a hope, not a mechanism. This is the story of how we turned it into a mechanism, and what else fell out when we did.

Flat lists fail silently

Our early marketplace snapshot was small compared with the full collection of first-party and third-party skills available in a working session. Even then, a flat list of names did not explain which copy was canonical or which capabilities overlapped. This is a historical account of that build; the current marketplace inventory is maintained in its generated catalog.

At that scale we kept hitting the same class of bug wearing different costumes. Two versions of one plugin got enabled at the same time, so every overlapping hook fired twice on every matching tool call. A skill existed at both user level and plugin level, and the two copies competed for the same invocation. Three different routing mechanisms grew up independently (a bash hook that grepped prompts for keywords, a skill-finding skill, and a routing skill), each answering "which capability applies here" from its own hand-maintained source.

Notice what these failures have in common. None of them is about a skill being bad at its job. Every one is about how skills relate to each other: this one duplicates that one, this one is a fork of that one, this one replaced that one, these two fire on the same trigger. A flat namespace has no way to say any of that. The relationships lived in our heads, in prose files, and in commit messages, which is to say they drifted.

Graph databases are having a moment for exactly this reason. Component registries and code maps already model software as nodes and edges because "what depends on what" is the question that matters. We decided the skill library deserved the same treatment.

What the graph actually is

Nothing exotic. A build script walks the repo and emits one JSON file: skills, plugins, commands, hooks, tags, MCP servers, and external upstreams as nodes, and a dozen typed edges carrying the relationships we kept failing to track: contains, fork-of, supersedes, overlaps-with, depends-on, replaces, extends, precedes, follows, augments, remediates, and usage-cooccurs. Some are structural, some are declared intent, some are mined from usage logs, and every edge records which of those it is. The newer types earn their keep at specific moments, which we'll get to.

Two design choices did most of the work.

Every edge records where it came from. A fork-of edge knows it was parsed from the provenance ledger; a tag edge knows it came from skill frontmatter; a usage edge knows it came from the monitor. When an edge is wrong, you know which source to fix.

The tag vocabulary is closed. Skills declare topics and stacks in frontmatter, and the compiler validates those values against a curated catalog. An unknown slug fails validation rather than silently becoming a routing signal. The catalog can grow through review, but a spelling mistake should not create a new category on its own.

The build is deterministic, byte for byte. Run it twice, get identical files. That let us add a staleness check: our weekly audit rebuilds the map, and if the committed artifact doesn't match the sources, the audit commits the fix and stops. A stale map is the fork problem all over again, so the map is not allowed to be stale.

Generated or nothing

The strongest argument for "generated, never hand-maintained" came from our own code, the same day we wrote it.

During the cleanup review of this very build, a reviewer found that our test fixture validator was a verbatim copy of the real validator, about 85 lines duplicated into a second file. By the time the review ran, the two copies had already diverged: the real one had gained a duplicate-id check the fixture copy lacked. Our fixtures were being validated by a weaker validator than the one guarding CI.

Read that timeline again. We were mid-project, actively building a system whose entire thesis is "copies drift and nobody notices," and our own copy drifted before the day was out. Not weeks. Hours. If you take one thing from this article, take that: the drift rate of duplicated logic is much faster than anyone's intuition, including ours, and we were primed to look for it.

The graph is generated from declared source material rather than edited as a standalone inventory. Some inputs still require curation: the tag vocabulary and non-derivable relationships, for example. Validate those inputs, record where each edge came from and regenerate the output. Generated documentation can then report the same inventory instead of maintaining a second list by hand.

What it unlocked day to day

The first consumer was routing. Our earlier prompt-time suggester used hand-written keyword patterns that needed editing as the library changed. The graph-based replacement can use declared names, tags and relationships from the same generated inventory. Its operating principle is restraint: a suggestion should require enough evidence to be useful, and a tool failure should not flood a session with unrelated advice. Actual runtime cost needs measurement on the installed version.

The second consumer was session startup. Four of our plugins used to print unconditional banners into every session: here are my commands, here is my context. Useful the first time, pure context tax the thousandth. They are gone, replaced by one hook that fingerprints the repo you are actually in (a Next.js config here, an Obsidian vault there) and surfaces up to eight skills relevant to that stack. Extends edges make the disclosure progressive: when a base skill and its specializations all match, the hook surfaces the base with a note that deeper skills exist, rather than spending its whole budget on one family. On repos where nothing matches, it prints nothing. Running it on the marketplace repo itself produces silence, which is correct, and honestly the silence took discipline to ship. Every instinct says print something.

The right skill at the right moment

Once routing and disclosure worked, a pattern became visible: a skill library doesn't have one moment where a suggestion helps, it has four. Session start (what's relevant to this repo). Prompt time (what's relevant to this request). After a skill finishes (what usually comes next). And when something breaks (what fixes this). Each moment wants a different relationship, and trying to serve all four from topic tags alone is how routers get noisy.

The subtlest addition was augments, for skills that improve a whole category of other skills rather than any one of them. Our humanizer skill makes any content-writing output sound less machine-generated. That's not a specialization of a content skill, and listing pairwise edges to every content skill would drift the moment a new one lands. So augments targets the category itself: one edge to the content-authoring tag covers every current and future member, including skills in plugins we don't own. When the router surfaces a content skill, it can mention the modifier that makes the output better.

For the something-broke moment, we added a third kind of tag: conditions. A condition like build-failure or type-error is a small curated entry that carries detection patterns, regexes matched against failing tool output rather than against anything the user typed. Skills declare which conditions they remediate, and a hook watches failed commands and suggests the top remediator. The satisfying part is who gets suggested: the best build-fixers in our ecosystem are third-party agents from the ecc plugin. The same map that guards against duplicating third-party capability now routes to it when it's the right tool. That's what complementary actually means.

And for the moment after a skill finishes, succession: a declared precedes edge for real pipelines (our PRD-writing command is followed by the PRD-grilling command, by design), backed by a mined follows edge derived from session logs when two skills keep occurring in order across sessions. Run the first step of a workflow and the hook offers the second. All four surfaces share one discipline, because it earned its keep in the router: at most one suggestion, silence otherwise, and no graph traversal at runtime.

The graph said no

The moment that convinced us this was more than plumbing came mid-build, from one of the agents doing the work.

Our implementation plan said: six hook scripts were left stranded in the wrong plugin by an old migration, move them to their proper home. The agent assigned to that task did something better than follow instructions. It diffed the six scripts against the target plugin first, and found that four of them already had live, wired, documented counterparts there. Two of those counterparts were byte-identical to the "stranded" copies. Two were newer. The plan's premise was wrong: those four were not stragglers waiting to be migrated, they were stale duplicates of a migration that had already happened. Moving them would have created a third copy of each.

The agent stopped, laid out the evidence, and proposed retiring the duplicates instead. That is the curation rule being enforced by relationship data rather than by vigilance. It is also, not coincidentally, exactly the failure mode the graph's fork-of and replaces edges exist to model. We shipped the retirement.

The same rule now runs at the front door. Our skill-ingest tool, Skill Forge, takes the map as an input and checks a candidate skill's name and description against every skill already in the graph before promotion. Strong overlap holds the candidate for review, with one deliberate exception: a candidate that declares it extends an existing skill is claiming layering, not duplication, and the gate turns that one pair into a reviewer's note instead of a block. Without the extends edge, the gate we built to stop duplicates would eventually punish exactly the deepening we want. "Does an enabled plugin already ship this?" used to be a question someone remembered to ask during review. Now it is a query.

One boundary we drew in the drift checker deserves a mention because it is easy to get wrong. The provenance ledger that records where each fork came from also contains prose describing how to check each fork for drift. The checker treats every bit of that as data. It reads files, fetches upstream content, hashes bytes, and compares. It never executes anything the ledger says, and a test pins that invariant. If you build anything that walks a graph of third-party references, make that rule structural, not behavioral.

Different is not the same as unreviewed

The drift checker's first real run embarrassed us. We pointed all seven forks at their upstream repos and every single one came back "drifted." Technically true. Practically useless. The only difference, in every case, was the five-line block of tags we had deliberately added : our own improvement, sitting in version control, flagged as if the upstream had moved underneath us. A two-way comparison, our file now against their file now, cannot tell those two situations apart. And a check that cries wolf seven times a week is a check people stop reading.

The fix was to give the checker a memory. When we ingest or review a fork, we record a baseline: the hash of the upstream as of the last time a human actually looked at it, stored as plain data in the same provenance ledger that records where the fork came from. The build also computes a normalized hash of our local copy with the injected tag block stripped out, so the one edit we made on purpose is invisible to the comparison.

Two comparisons instead of one yields four verdicts instead of two. Local matches baseline, upstream matches baseline: in sync, say nothing. We changed and upstream didn't: local-only : that is our work, it is in git, stay quiet. Upstream changed and we didn't: upstream-moved : the exact situation the whole system exists to catch, queue it for review. Both changed: diverged, queue that too. Only the last two page a human.

Re-baselining is a review decision: inspect the upstream change, adopt or decline it, and record the new reviewed baseline. The comparison should then distinguish intentional local changes from a newly moved upstream. That is the useful behavior to test, rather than measuring success by how quiet the report becomes.

The usage layer we didn't have to instrument

Existing usage logs can support an initial co-occurrence analysis without recording prompt text in the generated map. First verify how sessions and subagents are identified, then count only events that share the intended unit of analysis. A co-occurrence edge says that two skills were observed together; it does not prove that one helped the other.

The more useful metric turned out to be a structural one we named routing misses: skills that get used but that the router could never have suggested, because they have no tag edges to match on. That list feeds our refinement queue with a new kind of fix. Sometimes the right change is not to the skill at all. It is to the map.

Then we measured it

Everything above is argued from incidents : persuasive, but anecdotes. So the last thing we built was an eval suite, and the ground truth came from the map's own philosophy: mine it, don't author it. Every prompt in our session history that was followed by a real skill invocation is a labeled routing example. We restricted the mining to sessions from before the router went live, so it couldn't be graded on ground truth it helped create, and we resurrected the retired grep suggester from git history to serve as the baseline.

The first routing evaluation was a small development check, not a representative benchmark. It compared the retired keyword suggester with the map-based router using examples mined from earlier sessions. That design can expose obvious misses, but it does not establish general accuracy, latency or productivity. We have omitted the earlier headline percentages here because the source-bound run receipt was not available for this editorial refresh. To make a future result interpretable, publish the sample, candidate skill set, scoring rule, abstentions and held-out results together.

The best result wasn't a number. Writing the curation-gate regression test : the five historical duplicates that started this whole project must be flagged, forever : meant cutting test fixtures from the real generated artifact instead of writing them by hand. The first run against real data found the gate had been reading a field no real artifact sets. It had been silently matching nothing since the day it shipped, and every hand-built test fixture passed anyway, because hand-built fixtures encode the author's assumptions, not the artifact's reality. That's the same lesson this article keeps paying for: the copy you maintain by hand lies to you. The gate is fixed, and all five duplicates now get caught : including the four that required checking against the third-party ecosystem inventory, not just our own catalog.

A routing evaluation also needs to distinguish a suggestion being emitted, a skill being invoked and a user accepting the result. Those are separate observations. A scheduled report can track each available signal, but a missing suggestion or acceptance log must remain unavailable. It cannot be reconstructed from invocation counts alone.

What we didn't build

The core map is a generated JSON artifact rather than a requirement to adopt a graph database. Runtime consumers can use precomputed indexes, while maintenance tools ask named questions such as what extends a skill or which skills overlap. A larger graph engine is an option when the queries require it. Whatever storage you choose, keep provenance and the distinction between observed usage and inferred usefulness. An invocation log alone cannot tell you which suggestions were ignored.

If you run an agent skill library

Start with a narrow audit: export the installed inventory, identify the canonical source of each skill, compare likely duplicates, and record intentionally different forks. Then generate the relationships and fail validation when a declared source cannot be resolved.

Test the result against known failures before wiring it into prompt-time routing. Include a duplicate that should be caught, a legitimate specialization that should remain available, an upstream change that should prompt review, and an unrelated request that should receive no suggestion. Review drift before changing a recorded baseline.

Past a few dozen skills, your failures stop being about individual skills and start being about relationships between them. Model the relationships explicitly, in a generated artifact with provenance on every edge, and make staleness a build failure rather than a discovery. Close your tag vocabulary, because in generated data a typo is corruption. Prefer silence to noise everywhere your tooling talks to the model. And if part of your library is forked from somewhere, either diff it on a schedule or delete it, because the copy is already drifting, and it drifts faster than you think. Ours didn't last the day.

Sources and implementation references

Skill-map architecture: https://github.com/Rhize-Media/rhize-plugins/blob/main/docs/skill-map.md