Misclassification bounds for PAC-Bayesian sparse deep learning

Abstract

Recently, there has been a significant focus on exploring the theoretical aspects of deep learning, especially regarding its performance in classification tasks. Bayesian deep learning has emerged as a unified probabilistic framework, seeking to integrate deep learning with Bayesian methodologies seamlessly. However, there exists a gap in the theoretical understanding of Bayesian approaches in deep learning for classification. This study presents an attempt to bridge that gap. By leveraging PAC-Bayes bounds techniques, we present theoretical results on the prediction or misclassification error of a probabilistic approach utilizing Spike-and-Slab priors for sparse deep learning in classification. We establish non-asymptotic results for the prediction error. Additionally, we demonstrate that, by considering different architectures, our results can achieve minimax optimal rates in both low and high-dimensional settings, up to a logarithmic factor. Moreover, our additional logarithmic term yields slight improvements over previous works. Additionally, we propose and analyze an automated model selection approach aimed at optimally choosing a network architecture with guaranteed optimality.

arXiv admin note: text overlap with arXiv:1908.04847 by other authors

References in corpus (7)

• Spectrally-normalized margin bounds for neural networks
• User-friendly introduction to PAC-Bayes bounds
• Model Selection in Bayesian Neural Networks via Horseshoe Priors
• Efficient Variational Inference for Sparse Deep Learning with Theoretical Guarantee
• Masked Bayesian Neural Networks : Theoretical Guarantee and its Posterior Inference
• Minimax optimal high-dimensional classification using deep neural networks
• Asymptotic Properties for Bayesian Neural Network in Besov Space