3 papers
cs.LG2026
Lifelong In-Context Learning with Transformers Requires Parametric Forms of Attention
Luke McDermott, Robert W. Heath, Rahul Parhi
Lifelong continual learning remains an obstacle on the path to human-like intelligence. Modern transformers show sparks of intelligence with in-context learning. The quadratic natu…
cs.CL2026
LoLA: Low-Rank Linear Attention With Sparse Caching
Luke McDermott, Robert W. Heath, Rahul Parhi
The per-token cost of transformer inference scales with context length, preventing its application to lifelong in-context learning. Linear attention is an efficient alternative tha…
cs.LG2025
Finding Stable Subnetworks at Initialization with Dataset Distillation
Luke McDermott, Rahul Parhi
Recent works have shown that Dataset Distillation, the process for summarizing the training data, can be leveraged to accelerate the training of deep learning models. However, its…