Horseshoe Regularization for Machine Learning in Complex and Deep Models

Abstract

Since the advent of the horseshoe priors for regularization, global-local shrinkage methods have proved to be a fertile ground for the development of Bayesian methodology in machine learning, specifically for high-dimensional regression and classification problems. They have achieved remarkable success in computation, and enjoy strong theoretical support. Most of the existing literature has focused on the linear Gaussian case; see Bhadra et al. (2019b) for a systematic survey. The purpose of the current article is to demonstrate that the horseshoe regularization is useful far more broadly, by reviewing both methodological and computational developments in complex models that are more relevant to machine learning applications. Specifically, we focus on methodological challenges in horseshoe regularization in nonlinear and non-Gaussian models; multivariate models; and deep neural networks. We also outline the recent computational developments in horseshoe shrinkage for complex models along with a list of available software implementations that allows one to venture out beyond the comfort zone of the canonical linear regression problems.

References in corpus (18)

• Auto-Encoding Variational Bayes
• Needles and straw in haystacks: Empirical Bayes estimates of possibly sparse sequences
• Bayesian Compression for Deep Learning
• A simple sampler for the horseshoe estimator
• Needles and Straw in a Haystack: Posterior concentration for possibly sparse sequences
• Contributed Discussion to Uncertainty Quantification for the Horseshoe by Stéphanie van der Pas, Botond Szabó and Aad van der Vaart
• Quality of Uncertainty Quantification for Bayesian Neural Network Inference
• Simultaneous Variable and Covariance Selection with the Multivariate Spike-and-Slab Lasso
• GPU-accelerated Gibbs sampling: a case study of the Horseshoe Probit model
• Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priors
• Dropout as a Structured Shrinkage Prior
• Bayes Shrinkage at GWAS scale: Convergence and Approximation Theory of a Scalable MCMC Algorithm for the Horseshoe Prior
• Parameter expansion in local-shrinkage models
• Prior-preconditioned conjugate gradient method for accelerated Gibbs sampling in "large $n$ & large $p$" Bayesian sparse regression
• Bayesian Estimation of Gaussian Graphical Models with Predictive Covariance Selection
• Horseshoe Regularization for Feature Subset Selection
• Joint Mean-Covariance Estimation via the Horseshoe with an Application in Genomic Data Analysis
• Global-Local Mixtures

Cited by in corpus (1)

• Merging Two Cultures: Deep and Statistical Learning