Component Replacement in LLM Agents: A Critical Scoping Review of Task Benefit and Attribution

Abstract

Background: Choosing an agent controller and deciding which component to improve require different evidence. A replacement can change both the decision rule and what the agent can do after that decision. Objectives: We ask what existing replacement comparisons establish about task benefit and about attribution to local decision quality. Methods: This critical scoping review maps 348 studies and applies a five-step interpretation procedure to 94 comparison and diagnostic records from 42 studies, reading every inconsistent record against its source. Eight purposively selected cases illustrate the procedure. Results: The records comprise 33 executed task-outcome contrasts, 49 comparisons with restricted endpoints, and 12 diagnostic summaries. Only 14 task contrasts report contrast-level uncertainty, and the response policy is documented as retained in nine, all rule replacements from four studies; none of the 12 necessity-decision records executes a task contrast. Source checks distinguish whole-plan correctness, correct skill loading, and choice accuracy conditional on presentation from completed workflows. In Outcome Monitors, a count-matched timing placebo detects no timing effect (+0.88 points, 95% CI -4.39 to 6.14), so the source evidence attributes the detectable package gain to recovery content; detector selection at a fixed warning quota remains untested. Conclusions: The analysis separates evidence that can inform adoption from evidence that supports component development. Recovery and disruption, intervention timing, and downstream consumption organize three candidate mechanisms, and eight conditional reporting items specify the comparisons needed to distinguish the remaining explanations.

arXiv comment39 pages, 3 figures; the authors contributed equally