SkillEval: Decomposing Agent Skill Quality into Interpretable Signals

Abstract

Agent skills provide reusable knowledge that helps agents solve specialized tasks. As their use expands, evaluating skill quality becomes increasingly important. Existing evaluations often measure skill quality based on improvements in agent performance on specific tasks. However, a reusable skill may apply across multiple scenarios. Evaluation on a single downstream task mainly reflects the compatibility between a skill and the evaluated task, provides only a partial view of skill quality, and does not identify which aspects need improvement. We find that general properties of the \texttt{SKILL.md} document play an important role in skill quality. To evaluate these properties, we propose \textbf{SkillEval}, an interpretable framework for document-level skill evaluation. SkillEval produces interpretable scores for each property using a fixed, inspectable scoring direction. It further measures and reduces the influence of unrelated document features, so each score more specifically captures its intended semantic property. We use SkillEval to evaluate skills in controlled quality tests, showing that it reliably distinguishes skills of different quality. Its scores closely reflect downstream task performance, providing an early indication of skill effectiveness. We further explore SkillEval for diagnosing document weaknesses and guiding targeted revisions that improve the targeted properties and increase downstream task pass rates.