Misclassification bounds for PAC-Bayesian sparse deep learning
arXiv:2405.01304 · doi:10.1007/s10994-024-06690-0
Abstract
Recently, there has been a significant focus on exploring the theoretical aspects of deep learning, especially regarding its performance in classification tasks. Bayesian deep learning has emerged as a unified probabilistic framework, seeking to integrate deep learning with Bayesian methodologies seamlessly. However, there exists a gap in the theoretical understanding of Bayesian approaches in deep learning for classification. This study presents an attempt to bridge that gap. By leveraging PAC-Bayes bounds techniques, we present theoretical results on the prediction or misclassification error of a probabilistic approach utilizing Spike-and-Slab priors for sparse deep learning in classification. We establish non-asymptotic results for the prediction error. Additionally, we demonstrate that, by considering different architectures, our results can achieve minimax optimal rates in both low and high-dimensional settings, up to a logarithmic factor. Moreover, our additional logarithmic term yields slight improvements over previous works. Additionally, we propose and analyze an automated model selection approach aimed at optimally choosing a network architecture with guaranteed optimality.
arXiv commentarXiv admin note: text overlap with arXiv:1908.04847 by other authors
References in corpus (18)
- A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks
- Bayesian linear regression with sparse priors
- Spectrally-normalized margin bounds for neural networks
- Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity
- Sparse Regression Learning by Aggregation and Langevin Monte-Carlo
- Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality
- User-friendly introduction to PAC-Bayes bounds
- Model Selection in Bayesian Neural Networks via Horseshoe Priors
- 1-bit Matrix Completion: PAC-Bayesian Analysis of a Variational Approximation
- Understanding Priors in Bayesian Neural Networks at the Unit Level
- On the minimax optimality and superiority of deep neural network learning over sparse parameter spaces
- Deep Neural Networks Learn Non-Smooth Functions Effectively
- Convergence Rates of Variational Inference in Sparse Deep Learning
- Exponential weights in multivariate regression and a low-rankness favoring prior
- Efficient Variational Inference for Sparse Deep Learning with Theoretical Guarantee
- Masked Bayesian Neural Networks : Theoretical Guarantee and its Posterior Inference
- Minimax optimal high-dimensional classification using deep neural networks
- Asymptotic Properties for Bayesian Neural Network in Besov Space