On the generalization of bayesian deep nets for multi-class classification
arXiv:2002.09866
Abstract
Generalization bounds which assess the difference between the true risk and the empirical risk have been studied extensively. However, to obtain bounds, current techniques use strict assumptions such as a uniformly bounded or a Lipschitz loss function. To avoid these assumptions, in this paper, we propose a new generalization bound for Bayesian deep nets by exploiting the contractivity of the Log-Sobolev inequalities. Using these inequalities adds an additional loss-gradient norm term to the generalization bound, which is intuitively a surrogate of the model complexity. Empirically, we analyze the affect of this loss-gradient norm term using different deep nets.
References in corpus (13)
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
- Weight Uncertainty in Neural Networks
- Understanding deep learning requires rethinking generalization
- Train faster, generalize better: Stability of stochastic gradient descent
- Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
- Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks
- Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
- Fixup Initialization: Residual Learning Without Normalization
- A PAC-Bayesian Tutorial with A Dropout Bound
- PAC-Bayesian Theory Meets Bayesian Inference
- Horizon-Independent Optimal Prediction with Log-Loss in Exponential Families
- A Tight Excess Risk Bound via a Unified PAC-Bayesian-Rademacher-Shtarkov-MDL Complexity