Phase transitions in the mini-batch size for sparse and dense two-layer neural networks
arXiv:2305.06435 · doi:10.1088/2632-2153/ad1de6
Abstract
The use of mini-batches of data in training artificial neural networks is nowadays very common. Despite its broad usage, theories explaining quantitatively how large or small the optimal mini-batch size should be are missing. This work presents a systematic attempt at understanding the role of the mini-batch size in training two-layer neural networks. Working in the teacher-student scenario, with a sparse teacher, and focusing on tasks of different complexity, we quantify the effects of changing the mini-batch size . We find that often the generalization performances of the student strongly depend on and may undergo sharp phase transitions at a critical value , such that for the training process fails, while for the student learns perfectly or generalizes very well the teacher. Phase transitions are induced by collective phenomena firstly discovered in statistical mechanics and later observed in many fields of science. Observing a phase transition by varying the mini-batch size across different architectures raises several questions about the role of this hyperparameter in the neural network learning process.
arXiv comment15 pages, 8 figures
References in corpus (32)
- Parallel Tempering: Theory, Applications, and New Perspectives
- A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay
- Don't Decay the Learning Rate, Increase the Batch Size
- Statistical physics of inference: Thresholds and algorithms
- Information-theoretic thresholds from the cavity method
- Learning curves of generic features maps for realistic datasets with a teacher-student model
- The backtracking survey propagation algorithm for solving random K-SAT problems
- Out-of-equilibrium dynamical mean-field equations for the perceptron model
- Information-theoretic thresholds for community detection in sparse networks
- Equilibrium and non-Equilibrium regimes in the learning of Restricted Boltzmann Machines
- Critical jammed phase of the linear perceptron
- Unveiling the structure of wide flat minima in neural networks
- Information-theoretic and algorithmic thresholds for group testing
- Entropic gradient descent algorithms and wide flat minima
- Learning through atypical "phase transitions" in overparameterized neural networks
- Chaos and correlated avalanches in excitatory neural networks with synaptic plasticity
- Entropy production of a Brownian ellipsoid in the overdamped limit
- Learning curves for the multi-class teacher-student perceptron
- Solving non-linear Kolmogorov equations in large dimensions by using deep learning: a numerical comparison of discretization schemes
- Hard Optimization Problems have Soft Edges
- Ergodic observables in non-ergodic systems: the example of the harmonic chain
- Deep learning via message passing algorithms based on belief propagation
- Revisiting the Challenges of MaxClique
- Diffusion of a Brownian ellipsoid in a force field
- Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks
- Exact value for the average optimal cost of bipartite traveling-salesman and 2-factor problems in two dimensions
- Advective-diffusive motion on large scales from small-scale dynamics with an internal symmetry
- Two-Loop Corrections to Large Order Behavior of Theory
- Fluctuations in the random-link matching problem
- Criticality and conformality in the random dimer model
- Proliferation of non-linear excitations in the piecewise-linear perceptron
- Learning time-scales in two-layers neural networks