Posterior Concentration for Sparse Deep Learning

Abstract

Spike-and-Slab Deep Learning (SS-DL) is a fully Bayesian alternative to Dropout for improving generalizability of deep ReLU networks. This new type of regularization enables provable recovery of smooth input-output maps with unknown levels of smoothness. Indeed, we show that the posterior distribution concentrates at the near minimax rate for $\alpha$-Hölder smooth maps, performing as well as if we knew the smoothness level $\alpha$ ahead of time. Our result sheds light on architecture design for deep neural networks, namely the choice of depth, width and sparsity level. These network attributes typically depend on unknown smoothness in order to be optimal. We obviate this constraint with the fully Bayes construction. As an aside, we show that SS-DL does not overfit in the sense that the posterior concentrates on smaller networks with fewer (up to the optimal number of) nodes and links. Our results provide new theoretical justifications for deep ReLU networks from a Bayesian point of view.

Cited by in corpus (16)

• Uncertainty Quantification for Sparse Deep Learning
• Spike and slab variational Bayes for high dimensional logistic regression
• Convergence Rates of Variational Inference in Sparse Deep Learning
• Layer Adaptive Node Selection in Bayesian Neural Networks: Statistical Guarantees and Implementation Details
• Fiedler Regularization: Learning Neural Networks with Graph Sparsity
• Consistent Sparse Deep Learning: Theory and Computation
• Deep Learning Partial Least Squares
• Sparse Deep Learning: A New Framework Immune to Local Traps and Miscalibration
• Variable Selection with Rigorous Uncertainty Quantification using Deep Bayesian Neural Networks: Posterior Concentration and Bernstein-von Mises Phenomenon
• Approximate Bayesian Computation via Classification
• Minimax optimal high-dimensional classification using deep neural networks
• Bayesian Probabilistic Numerical Integration with Tree-Based Models
• Posterior contraction for deep Gaussian process priors
• Adaptive Variational Bayesian Inference for Sparse Deep Neural Network
• Merging Two Cultures: Deep and Statistical Learning
• HALO: Learning to Prune Neural Networks with Shrinkage