On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implications

Abstract

This paper follows up on a recent work of Neu et al. (2021) and presents some new information-theoretic upper bounds for the generalization error of machine learning models, such as neural networks, trained with SGD. We apply these bounds to analyzing the generalization behaviour of linear and two-layer ReLU networks. Experimental study of these bounds provide some insights on the SGD training of neural networks. They also point to a new and simple regularization scheme which we show performs comparably to the current state of the art.

arXiv commentAccepted to ICLR2022

References in corpus (18)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Reconciling modern machine learning practice and the bias-variance trade-off
• A Convergence Theory for Deep Learning via Over-Parameterization
• A Stochastic Gradient Method with an Exponential Convergence Rate for Finite Training Sets
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
• Three Factors Influencing Minima in SGD
• Information-theoretic analysis of generalization capability of learning algorithms
• Sharp Minima Can Generalize For Deep Nets
• Generalization in Adaptive Data Analysis and Holdout Reuse
• Why gradient clipping accelerates training: A theoretical justification for adaptivity
• Norm-Based Capacity Control in Neural Networks
• A PAC-Bayesian Analysis of Randomized Learning with Application to Stochastic Gradient Descent
• Learners that Use Little Information
• Stability and Convergence Trade-off of Iterative Optimization Algorithms
• Sharpened Generalization Bounds based on Conditional Mutual Information and an Application to Noisy, Iterative Algorithms
• On Random Subset Generalization Error Bounds and the Stochastic Gradient Langevin Dynamics Algorithm
• A Study of Gradient Variance in Deep Learning