7 papers
To Retain or to Adapt? Generalizing Continual Learning
Giulia Lanzillotta, Mandana Samiei, Doina Precup +2
The Continual Learning (CL) literature has long been driven by the goal of mitigating catastrophic forgetting. This objective rests on a pervasive, often unstated assumption: that…
When One Adapter Speaks for Many: Discovering Low-Rank Redundancy in Continual Fine-Tuning
Tanguy Dieudonné, Giulia Lanzillotta, Enis Simsar +2
Low-Rank Adaptation (LoRA) has become the standard tool for parameter-efficient fine-tuning of large pretrained models. When applied sequentially across tasks in Continual Learning…
Barriers for Learning in an Evolving World: Mathematical Understanding of Loss of Plasticity
Amir Joudaki, Giulia Lanzillotta, Mohammad Samragh Razlighi +5
Deep learning models excel in stationary data but struggle in non-stationary environments due to a phenomenon known as loss of plasticity (LoP), the degradation of their ability to…
Heads collapse, features stay: Why Replay needs big buffers
Giulia Lanzillotta, Damiano Meier, Thomas Hofmann
A persistent paradox in continual learning (CL) is that neural networks often retain linearly separable representations of past tasks even when their output predictions fail. We fo…
Revisiting Knowledge Distillation: The Hidden Role of Dataset Size
Giulia Lanzillotta, Felix Sarnthein, Gil Kur +2
The concept of knowledge distillation (KD) describes the training of a student model from a teacher model and is a widely adopted technique in deep learning. However, it is still n…
The Importance of Being Lazy: Scaling Limits of Continual Learning
Jacopo Graldi, Alessandro Breccia, Giulia Lanzillotta +2
Despite recent efforts, neural networks still struggle to learn in non-stationary environments, and our understanding of catastrophic forgetting (CF) is far from complete. In this…