6 papers
BISECT: Bilevel Spectral Decomposition for Low-Rank Continual Learning
Tom Saliencro, Wen-Hsuan Liu, Rohan Desai +2
Exemplar-free continual learning on pre-trained Vision Transformers increasingly relies on Low-Rank Adaptation (LoRA), yet prevailing methods govern the low-rank update space throu…
RASP: Read-All, Sparse-Plasticity Continual Learning in a Stationary-Size LoRA
Wen-Hsuan Liu, Amara Okonkwo, Tom Saliencro +2
Parameter-efficient continual learning on pre-trained Vision Transformers largely relies on a costly premise: preventing catastrophic forgetting requires either spawning a dedicate…
Score the Update, Not the Token: Descent-Aligned Routing for Combinatorial LoRA Experts
Priya Nair, Lukas Brenner, Maya Lindqvist +5
Mixture-of-LoRA-experts methods raise the capacity of low-rank adaptation by routing each token to a few low-rank experts. Nearly all of them tie one input-side factor to one outpu…
Uncertainty Is Not Enough: Value-of-Information Routing for Mixtures of LoRA Experts
Tom Saliencro, Rohan Desai, Priya Nair +2
Mixtures of low-rank adaptation experts increase parameter-efficient capacity by routing each input through a subset of adapters. Recent dynamic routers activate more experts when…
Spend Experts Where You Are Unsure: Confidence-Adaptive Routing for Mixture-of-Experts LoRA
Tom Saliencro, Rohan Desai, Priya Nair +2
Mixture-of-Experts (MoE) variants of Low-Rank Adaptation (LoRA) route every token to a fixed number of experts . Tokens differ in how uncertain the model is about them, so a sin…
FRAME: Learning the Adaptation Domain with a Mixture of Fractional-Fourier Experts
Tom Saliencro, Maya Lindqvist, Rohan Desai +2
Parameter-efficient fine-tuning (PEFT) reparameterizes weight updates in a fixed basis: low-rank adapters operate in the spatial domain, while a recent line of spectral methods ope…