Drowning in Routine: Signal Dilution in Multi-Turn Agent Training

Abstract

Multi-turn agents interleave consequential decisions with routine execution: some actions change the downstream return distribution, while others are necessary but reward-equivalent. The cost of trajectory-level credit assignment, often attributed to long horizons, is in fact governed by decision density $\ddot{I}$: the fraction of turns whose actions affect the return. When decision density is low, routine turns create signal dilution: they add gradient variance to trajectory-level estimators such as GRPO without adding expected signal. Under explicit assumptions, the resulting turn-level to trajectory-level signal-to-noise ratio scales as $\ddot{I}{}^{-1/2}$, provided critic error remains controlled. The same analysis identifies the complementary regime: at high decision density, trajectory-level methods can remain competitive while avoiding the cost of a critic. In a controlled environment where $\ddot{I}$ is exactly tunable, the predicted scaling is recovered with $R^2=0.999$, and the training-step gap widens significantly as $\ddot{I}\to 0$.

Accepted at the FAGEN Workshop at ICML 2026, Seoul, South Korea. 14 pages, 9 figures