210 citations · 1.5k across the 62 of their papers we have counts for
63 papers
Phase Transitions in Attention: A Bayesian Theory of Copy Head Emergence
Itay Lavie, Kirsten Fischer, Andrey Lekov +3
Attention is the key mechanism underlying in-context learning in transformers, and attention patterns have been observed empirically to emerge abruptly during training. We present…
Discrete signaling mediates chaotic regularization in recurrent neural networks
Jan Bauer, Christian Keup, Jonathan Kadmon +1
Cortical circuits operate in a regime of intrinsic chaos, where even tiny changes in input can lead to divergent neural responses. Yet, remarkably, population codes in the brain va…
A unified theory of feature learning in RNNs and DNNs
Jan P. Bauer, Kirsten Fischer, Moritz Helias +1
Recurrent and deep neural networks (RNNs/DNNs) are cornerstone architectures in machine learning. Remarkably, RNNs differ from DNNs only by weight sharing, as can be shown through…
Lecture notes: From Gaussian processes to feature learning
Moritz Helias, Javed Lindner, Lars Schutzeichel +1
These lecture notes develop the theory of learning in deep and recurrent neuronal networks from the point of view of Bayesian inference. The aim is to enable the reader to understa…
Dynamics of neural scaling laws in random feature regression with powerlaw-distributed kernel eigenvalues
Jakob Kramp, Javed Lindner, Moritz Helias
Training large neural networks exposes neural scaling laws for the generalization error, which points to a universal behavior across network architectures of learning in high dimen…
Reduction of interaction order in hard combinatorial optimization via conditionally independent degrees of freedom
Alexandru Ciobanu, David Dahmen, John Paul Strachan +1
Combinatorial optimization problems have a broad range of applications and map to physical systems with complex dynamics. Among them, the 3-SAT problem is prominent due to its NP-c…