ResCue: Residual Spatial Cueing for Language-Conditioned Imitation Learning

Abstract

Visual imitation policies must learn which image regions matter for action, whether applying the same operation to different objects or selecting a language-specified target. We introduce ResCue (Residual Spatial Cueing), which converts language-derived target information into a spatial condition for visual policy learning. Pretrained grounding produces a target mask; an independent cue encoder adds its features to spatial RGB features before further visual processing and pooling. This retains scene context while making the target explicit. With Diffusion Policy (DP), ResCue achieves 56.5% average success across 15 simulated tasks, versus 33.1% for DP and 40.% for the strongest baseline, ${\pi }_0$, with gains in both ordinary manipulation and instruction-dependent object selection. On three real-world bimanual tasks, it reaches 60.0%, compared with 31.7% for ${\pi }_0$. Matched-mask ablations favor residual fusion over input-level and pooled alternatives, while test-time interventions show that target-correct guidance matters for execution.