1 citations · 1 across the 10 of their papers we have counts for
5 papers · 1 filter
Heads collapse, features stay: Why Replay needs big buffers
Giulia Lanzillotta, Damiano Meier, Thomas Hofmann
A persistent paradox in continual learning (CL) is that neural networks often retain linearly separable representations of past tasks even when their output predictions fail. We fo…
Revisiting Knowledge Distillation: The Hidden Role of Dataset Size
Giulia Lanzillotta, Felix Sarnthein, Gil Kur +2
The concept of knowledge distillation (KD) describes the training of a student model from a teacher model and is a widely adopted technique in deep learning. However, it is still n…
Barriers for Learning in an Evolving World: Mathematical Understanding of Loss of Plasticity
Amir Joudaki, Giulia Lanzillotta, Mohammad Samragh Razlighi +5
Deep learning models excel in stationary data but struggle in non-stationary environments due to a phenomenon known as loss of plasticity (LoP), the degradation of their ability to…
Reactivation: Empirical NTK Dynamics Under Task Shifts
Yuzhi Liu, Zixuan Chen, Zirui Zhang +2
The Neural Tangent Kernel (NTK) offers a powerful tool to study the functional dynamics of neural networks. In the so-called lazy, or kernel regime, the NTK remains static during t…
The Importance of Being Lazy: Scaling Limits of Continual Learning
Jacopo Graldi, Alessandro Breccia, Giulia Lanzillotta +2
Despite recent efforts, neural networks still struggle to learn in non-stationary environments, and our understanding of catastrophic forgetting (CF) is far from complete. In this…