Do Language Models Need Sleep? Offline Recurrence for Improved Online Inference

Sangyun Lee, Sean McLeish, Tom Goldstein +1

Transformer-based large language models are increasingly used for long-horizon tasks; however, their attention mechanism scales poorly with context length. To handle this, we study…