A PAC-Bayesian Analysis of Randomized Learning with Application to Stochastic Gradient Descent

Abstract

We study the generalization error of randomized learning algorithms -- focusing on stochastic gradient descent (SGD) -- using a novel combination of PAC-Bayes and algorithmic stability. Importantly, our generalization bounds hold for all posterior distributions on an algorithm's random hyperparameters, including distributions that depend on the training data. This inspires an adaptive sampling algorithm for SGD that optimizes the posterior at runtime. We analyze this algorithm in the context of our generalization bounds and evaluate it on a benchmark dataset. Our experiments demonstrate that adaptive sampling can reduce empirical risk faster than uniform sampling while also improving out-of-sample accuracy.

In Neural Information Processing Systems (NIPS) 2017. The latest version specifies that the references to Kuzborskij & Lampert (2017) are for v2 of their manuscript, which was posted to arXiv in March, 2017. Importantly, Theorem 3 therein (a stability bound for convex losses) has a different form than the final version

Cited by in corpus (21)

• User-friendly introduction to PAC-Bayes bounds
• A Primer on PAC-Bayesian Learning
• High probability generalization bounds for uniformly stable algorithms with nearly optimal rate
• Tighter risk certificates for neural networks
• On Generalization Error Bounds of Noisy Gradient Methods for Non-Convex Learning
• PAC-Bayes with Backprop
• Generalization Error Bounds with Probabilistic Guarantee for SGD in Nonconvex Optimization
• Heavy Tails in SGD and Compressibility of Overparametrized Neural Networks
• Generalization bounds for deep learning
• Data-dependent PAC-Bayes priors via differential privacy
• Stability and Generalization of Stochastic Gradient Methods for Minimax Problems
• PAC-Bayes Bounds for Meta-learning with Data-Dependent Prior
• Adaptive Task Sampling for Meta-Learning
• Improved Learning Rates for Stochastic Optimization
• De-randomized PAC-Bayes Margin Bounds: Applications to Non-convex and Non-smooth Predictors
• On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implications
• PAC-Bayesian Generalization Bounds for MultiLayer Perceptrons
• Practical calibration of the temperature parameter in Gibbs posteriors
• Entropy-SGD optimizes the prior of a PAC-Bayes bound: Generalization properties of Entropy-SGD and data-dependent priors
• Generalization Bounds for Meta-Learning via PAC-Bayes and Uniform Stability
• Optimal Rates for Random Order Online Optimization