24 citations · 87 across the 13 of their papers we have counts for
5 papers · 1 filter
When Forgetting is not Catastrophic: On the Mechanics of Spurious Forgetting
Vedant Palit, Florent Draye, Nicolas Zucchet +2
Knowledge that a language model appears to forget during finetuning often remains stored and can be recovered, a phenomenon called spurious forgetting. Finetuning on new facts can…
Divergence controls entropy in distillation
Nicolas Zucchet, Scott W. Linderman
Distillation has become a core primitive of large language model training, but its properties are not yet well understood. We take an entropic perspective, studying how the entropy…
Switching Linear Attention
Hyun Dong Lee, Xavier Gonzalez, Nicolas Zucchet +3
Designing expressive sequence layers with efficient inference remains a central challenge in modern machine learning. Standard softmax attention achieves excellent sequence modelin…
Language models suffer from a curse of ambiguity
Nicolas Zucchet, Hyun Dong Lee, Scott Linderman
Large language models increasingly rely on sampling as a driver of their own improvement, making the fidelity of their learned distributions more critical than ever. Yet, not all d…
Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks
Tiberiu Musat, Tiago Pimentel, Nicolas Zucchet +1
We present a theoretical framework to explain the emergence of inductive reasoning abilities in Transformer language models. While previous works on Transformer learning dynamics h…