4 papers
Lifelong In-Context Learning with Transformers Requires Parametric Forms of Attention
Luke McDermott, Robert W. Heath, Rahul Parhi
Lifelong continual learning remains an obstacle on the path to human-like intelligence. Modern transformers show sparks of intelligence with in-context learning. The quadratic natu…
LoLA: Low-Rank Linear Attention With Sparse Caching
Luke McDermott, Robert W. Heath, Rahul Parhi
The per-token cost of transformer inference scales with context length, preventing its application to lifelong in-context learning. Linear attention is an efficient alternative tha…
Finding Stable Subnetworks at Initialization with Dataset Distillation
Luke McDermott, Rahul Parhi
Recent works have shown that Dataset Distillation, the process for summarizing the training data, can be leveraged to accelerate the training of deep learning models. However, its…
Embedding Compression for Efficient Re-Identification
Luke McDermott
Real world re-identfication (ReID) algorithms aim to map new observations of an object to previously recorded instances. These systems are often constrained by quantity and size of…