An Adaptive Empirical Bayesian Method for Sparse Deep Learning

Abstract

We propose a novel adaptive empirical Bayesian method for sparse deep learning, where the sparsity is ensured via a class of self-adaptive spike-and-slab priors. The proposed method works by alternatively sampling from an adaptive hierarchical posterior distribution using stochastic gradient Markov Chain Monte Carlo (MCMC) and smoothly optimizing the hyperparameters using stochastic approximation (SA). We further prove the convergence of the proposed method to the asymptotically correct distribution under mild conditions. Empirical applications of the proposed method lead to the state-of-the-art performance on MNIST and Fashion MNIST with shallow convolutional neural networks and the state-of-the-art compression performance on CIFAR10 with Residual Networks. The proposed method also improves resistance to adversarial attacks.

arXiv commentAccepted by NeurIPS 2019; Update the assumption on the regularity of Poisson equation

References in corpus (15)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Explaining and Harnessing Adversarial Examples
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Simulated Tempering: A New Monte Carlo Scheme
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Stochastic Gradient Hamiltonian Monte Carlo
• Bayesian Compression for Deep Learning
• User-friendly guarantees for the Langevin Monte Carlo with inaccurate gradient
• Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks
• On the Convergence of Stochastic Gradient MCMC Algorithms with High-Order Integrators
• Dropout Inference in Bayesian Neural Networks with Alpha-divergences
• Bayesian GAN
• Convex Optimization with Unbounded Nonconvex Oracles using Simulated Annealing