4 papers · 1 filter
Phase Transitions in Attention: A Bayesian Theory of Copy Head Emergence
Itay Lavie, Kirsten Fischer, Andrey Lekov +3
Attention is the key mechanism underlying in-context learning in transformers, and attention patterns have been observed empirically to emerge abruptly during training. We present…
Applications of Statistical Field Theory in Deep Learning
Zohar Ringel, Noa Rubin, Edo Mor +2
Deep learning algorithms have made incredible strides in the past decade, yet due to their complexity, the science of deep learning remains in its early stages. Being an experiment…
Demystifying Spectral Bias on Real-World Data
Itay Lavie, Zohar Ringel
Kernel ridge regression (KRR) and Gaussian processes (GPs) are fundamental tools in statistics and machine learning, with recent applications to highly over-parameterized deep neur…
Grokking as a First Order Phase Transition in Two Layer Networks
Noa Rubin, Inbar Seroussi, Zohar Ringel
A key property of deep neural networks (DNNs) is their ability to learn new features during training. This intriguing aspect of deep learning stands out most clearly in recently re…