End-to-End Test-Time Training for Long Context

Arnuv Tandon, Karan Dalal, Xinhao Li +11

We formulate long-context language modeling as a problem in continual learning rather than architecture design. Under this formulation, we only use a standard architecture -- a Tra…