Showing cs.LGShow all
3 papers · 1 filter
cs.LG2025
Dataset distillation for memorized data: Soft labels can leak held-out teacher knowledge
Freya Behrens, Lenka Zdeborová
Dataset distillation aims to compress training data into fewer examples via a teacher, from which a student can learn effectively. While its success is often attributed to structur…
cs.LG2025
Counting in Small Transformers: The Delicate Interplay between Attention and Feed-Forward Layers
Freya Behrens, Luca Biggio, Lenka Zdeborová
Next to scaling considerations, architectural design choices profoundly shape the solution space of transformers. In this work, we analyze the solutions simple transformer blocks i…
cs.LG2024
A phase transition between positional and semantic learning in a solvable model of dot-product attention
Hugo Cui, Freya Behrens, Florent Krzakala +2
Many empirical studies have provided evidence for the emergence of algorithmic mechanisms (abilities) in the learning of language models, that lead to qualitative improvements of t…