210 citations · 1.5k across the 62 of their papers we have counts for
4 papers · 1 filter
Phase Transitions in Attention: A Bayesian Theory of Copy Head Emergence
Itay Lavie, Kirsten Fischer, Andrey Lekov +3
Attention is the key mechanism underlying in-context learning in transformers, and attention patterns have been observed empirically to emerge abruptly during training. We present…
Applications of Statistical Field Theory in Deep Learning
Zohar Ringel, Noa Rubin, Edo Mor +2
Deep learning algorithms have made incredible strides in the past decade, yet due to their complexity, the science of deep learning remains in its early stages. Being an experiment…
Graph Neural Networks Do Not Always Oversmooth
Bastian Epping, Alexandre René, Moritz Helias +1
Graph neural networks (GNNs) have emerged as powerful tools for processing relational data in applications. However, GNNs suffer from the problem of oversmoothing, the property tha…
Speed Limits for Deep Learning
Inbar Seroussi, Alexander A. Alemi, Moritz Helias +1
State-of-the-art neural networks require extreme computational power to train. It is therefore natural to wonder whether they are optimally trained. Here we apply a recent advancem…