Ratchet: How Reliable Must an LLM Judge Be to Retire a Skill?

Abstract

A large language model (LLM) agent that writes and edits its own skill library must also decide which skills to keep, from one noisy scalar per skill. The answer is exact: a judge scoring failures as passes at rate $(1-\tau )/2$ or above retires nothing, at any sample size, for eviction margin $\tau$. Audits find that machinery is rarely built: LLM-written skills are worth $+0.0$ percentage points (pp) against a no-skill control, human-written ones $+16.2$pp. Unmaintained, a library enters \emph{library drift}, growing until injecting a skill scores worse than injecting nothing. \textbf{Ratchet} repairs this: it evicts each skill on its measured contribution, caps the library at width $C$, and constrains synthesis, lifting held-out $pass@1$ by $+0.328$ on a hard MBPP+ slice. The matching non-divergence bound is finite for exactly two reasons, $C$ and $\tau$. Our contribution is the condition this repair carries and no deployed system states. In reference-free domains the scalar comes from an LLM judge, whose two error directions, modelled as a binary channel, behave nothing alike. Passes scored as failures cost sample efficiency, which more trials buy back; failures scored as passes displace the eviction statistic, and no correction inside the rule recovers it. End-task score is a poor alarm, moving by at most a fifth of the governed lift and not monotonically in the rate. We prove both edges of the certifiable region, confirm them in a running loop, and place a judge on a known side in one offline pass.

Code: https://github.com/amazon-science/Self-Evolving-Agents-Ratchet