On-Demand Attention: Language Models Know When to Recall

Haibo Feng, Ruiqi Liang, Hanyang Peng +2

Reasoning and agentic workloads increasingly demand efficient long-context inference. Yet full-attention decoding reads the growing history at every step, although the benefit of g…