Optimizing Information-theoretical Generalization Bounds via Anisotropic Noise in SGLD

Abstract

Recently, the information-theoretical framework has been proven to be able to obtain non-vacuous generalization bounds for large models trained by Stochastic Gradient Langevin Dynamics (SGLD) with isotropic noise. In this paper, we optimize the information-theoretical generalization bound by manipulating the noise structure in SGLD. We prove that with constraint to guarantee low empirical risk, the optimal noise covariance is the square root of the expected gradient covariance if both the prior and the posterior are jointly optimized. This validates that the optimal noise is quite close to the empirical gradient covariance. Technically, we develop a new information-theoretical bound that enables such an optimization analysis. We then apply matrix analysis to derive the form of optimal noise covariance. Presented constraint and results are validated by the empirical observations.

arXiv commentAccepted by Neurips 2021

References in corpus (15)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Information-theoretic analysis of generalization capability of learning algorithms
• Generalization in Adaptive Data Analysis and Holdout Reuse
• Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approach
• The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects
• Data-Dependent Stability of Stochastic Gradient Descent
• Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimates
• Sharpened Generalization Bounds based on Conditional Mutual Information and an Application to Noisy, Iterative Algorithms
• On Generalization Error Bounds of Noisy Gradient Methods for Non-Convex Learning
• A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima
• An Empirical Study of Large-Batch Stochastic Gradient Descent with Structured Covariance Noise
• Dynamic of Stochastic Gradient Descent with State-Dependent Noise
• Information-Theoretic Generalization Bounds for Stochastic Gradient Descent