Wide flat minima and optimal generalization in classifying high-dimensional Gaussian mixtures

Abstract

We analyze the connection between minimizers with good generalizing properties and high local entropy regions of a threshold-linear classifier in Gaussian mixtures with the mean squared error loss function. We show that there exist configurations that achieve the Bayes-optimal generalization error, even in the case of unbalanced clusters. We explore analytically the error-counting loss landscape in the vicinity of a Bayes-optimal solution, and show that the closer we get to such configurations, the higher the local entropy, implying that the Bayes-optimal solution lays inside a wide flat region. We also consider the algorithmically relevant case of targeting wide flat minima of the (differentiable) mean squared error loss. Our analytical and numerical results show not only that in the balanced case the dependence on the norm of the weights is mild, but also, in the unbalanced case, that the performances can be improved.

19 pages, 4 figures. arXiv admin note: text overlap with arXiv:2006.07897

References in corpus (16)

• Proceedings of the 29th International Conference on Machine Learning (ICML-12)
• The structural glass transition and the entropy of the metastable states
• Recipes for metastable states in Spin Glasses
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Fantastic Generalization Measures and Where to Find Them
• Unreasonable Effectiveness of Learning Neural Networks: From Accessible States and Robust Ensembles to Basic Algorithmic Schemes
• Subdominant Dense Clusters Allow for Simple Learning and High Computational Performance in Neural Networks with Discrete Synapses
• Shaping the learning landscape in neural networks around wide flat minima
• Origin of the computational hardness for learning with binary synapses
• Properties of the geometry of solutions and capacity of multi-layer neural networks with Rectified Linear Units activations
• Local entropy as a measure for sampling solutions in Constraint Satisfaction Problems
• Statistical learning theory of structured data
• A Model of Double Descent for High-dimensional Binary Linear Classification
• Beyond the storage capacity: data driven satisfiability transition
• High Dimensional Classification via Regularized and Unregularized Empirical Risk Minimization: Precise Error and Optimal Loss
• Generalization from correlated sets of patterns in the perceptron

Cited by in corpus (4)

• Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized models
• Unveiling the structure of wide flat minima in neural networks
• Learning through atypical "phase transitions" in overparameterized neural networks
• Bias-variance decomposition of overparameterized regression with random linear features