A distributional simplicity bias in the learning dynamics of transformers

Riccardo Rende, Federica Gerace, Alessandro Laio +1

The remarkable capability of over-parameterised neural networks to generalise effectively has been explained by invoking a ``simplicity bias'': neural networks prevent overfitting…