When Privileged Guidance Misaligns: State-Matched Routing and Contextualized Self-Distillation for Multi-Turn Agents

Abstract

Multi-turn agents trained with sparse outcome rewards receive limited supervision for intermediate decisions. Privileged on-policy self-distillation provides denser guidance by having a synchronized teacher re-score student responses with training-only references. However, the student may take different actions or complete subgoals in another order, reaching states for which the reference provides no applicable continuation. The resulting state--reference mismatch can lead the reference-conditioned teacher to penalize valid actions. We introduce State-Matched Routing and Contextualized Self-Distillation (SMRC-SD), which uses the student's reached state to decide when and how a reference should guide the student. At each turn, SMRC-SD checks state compatibility with the reference, distills only on matched turns, and contextualizes the reference by augmenting it with a reached-state summary and a compatible continuation. At inference, the policy requires neither references nor state matching. Experiments spanning three model sizes show gains over unconditional full-path self-distillation in embodied interaction (ALFWorld), web navigation (WebShop), and tool use (FTRL). FTRL-trained policies also transfer to BFCL and ToolHop without additional training. Code is available at https://github.com/liujunzhuo/SMRC-SD.

arXiv commentUpdated the paper with Qwen3-4B experiments and FTRL tool-use benchmark