Generalisation under gradient descent via deterministic PAC-Bayes
arXiv:2209.02525
Abstract
We establish disintegrated PAC-Bayesian generalisation bounds for models trained with gradient descent methods or continuous gradient flows. Contrary to standard practice in the PAC-Bayesian setting, our result applies to optimisation algorithms that are deterministic, without requiring any de-randomisation step. Our bounds are fully computable, depending on the density of the initial distribution and the Hessian of the training objective over the trajectory. We show that our framework can be applied to a variety of iterative optimisation algorithms, including stochastic gradient descent (SGD), momentum-based schemes, and damped Hamiltonian dynamics.
References in corpus (20)
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Score-Based Generative Modeling through Stochastic Differential Equations
- Train faster, generalize better: Stability of stochastic gradient descent
- A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
- Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent
- Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
- A Note on the PAC Bayesian Theorem
- Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpoints
- Asymmetric Valleys: Beyond Sharp and Flat Local Minima
- Tighter risk certificates for neural networks
- Sharper bounds for uniformly stable algorithms
- Differentiable PAC-Bayes Objectives with Partially Aggregated Neural Networks
- Learning PAC-Bayes Priors for Probabilistic Neural Networks
- Generalization Bounds via Convex Analysis
- On Margins and Derandomisation in PAC-Bayes
- De-randomized PAC-Bayes Margin Bounds: Applications to Non-convex and Non-smooth Predictors
- Conditionally Gaussian PAC-Bayes
- PAC-Bayesian Transportation Bound
- Learning Stochastic Majority Votes by Minimizing a PAC-Bayes Generalization Bound
- Algorithmic Stability of Heavy-Tailed SGD with General Loss Functions