Neural networks trained with SGD learn distributions of increasing complexity

Maria Refinetti, Alessandro Ingrosso, Sebastian Goldt

The ability of deep neural networks to generalise well even when they interpolate their training data has been explained using various "simplicity biases". These theories postulate…