Variable Selection with Rigorous Uncertainty Quantification using Deep Bayesian Neural Networks: Posterior Concentration and Bernstein-von Mises Phenomenon

Abstract

This work develops rigorous theoretical basis for the fact that deep Bayesian neural network (BNN) is an effective tool for high-dimensional variable selection with rigorous uncertainty quantification. We develop new Bayesian non-parametric theorems to show that a properly configured deep BNN (1) learns the variable importance effectively in high dimensions, and its learning rate can sometimes "break" the curse of dimensionality. (2) BNN's uncertainty quantification for variable importance is rigorous, in the sense that its 95% credible intervals for variable importance indeed covers the truth 95% of the time (i.e., the Bernstein-von Mises (BvM) phenomenon). The theoretical results suggest a simple variable selection algorithm based on the BNN's credible intervals. Extensive simulation confirms the theoretical findings and shows that the proposed algorithm outperforms existing classic and neural-network-based variable selection methods, particularly in high dimensions.

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• TensorFlow: A system for large-scale machine learning
• A significance test for the lasso
• Group Sparse Regularization for Deep Neural Networks
• Breaking the Curse of Dimensionality with Convex Neural Networks
• Nonparametric sparsity and regularization
• Sparse-Input Neural Networks for High-dimensional Nonparametric Regression and Classification
• Model Selection in Bayesian Neural Networks via Horseshoe Priors
• Provable Bayesian Inference via Particle Mirror Descent
• Approximation and Estimation for High-Dimensional Deep Learning Networks
• Approximation spaces of deep neural networks
• Posterior Concentration for Sparse Deep Learning

Cited by in corpus (2)

• Uncertainty Quantification for Sparse Deep Learning
• Consistent Feature Selection for Analytic Deep Neural Networks