Sparsity information and regularization in the horseshoe and other shrinkage priors

Abstract

The horseshoe prior has proven to be a noteworthy alternative for sparse Bayesian estimation, but has previously suffered from two problems. First, there has been no systematic way of specifying a prior for the global shrinkage hyperparameter based on the prior information about the degree of sparsity in the parameter vector. Second, the horseshoe prior has the undesired property that there is no possibility of specifying separately information about sparsity and the amount of regularization for the largest coefficients, which can be problematic with weakly identified parameters, such as the logistic regression coefficients in the case of data separation. This paper proposes solutions to both of these problems. We introduce a concept of effective number of nonzero parameters, show an intuitive way of formulating the prior for the global hyperparameter based on the sparsity assumptions, and argue that the previous default choices are dubious based on their tendency to favor solutions with more unshrunk parameters than we typically expect a priori. Moreover, we introduce a generalization to the horseshoe prior, called the regularized horseshoe, that allows us to specify a minimum level of regularization to the largest values. We show that the new prior can be considered as the continuous counterpart of the spike-and-slab prior with a finite slab width, whereas the original horseshoe resembles the spike-and-slab with an infinitely wide slab. Numerical experiments on synthetic and real world data illustrate the benefit of both of these theoretical advances.

References in corpus (9)

• Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
• The No-U-Turn Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte Carlo
• A Conceptual Introduction to Hamiltonian Monte Carlo
• Needles and straw in haystacks: Empirical Bayes estimates of possibly sparse sequences
• Comparison of Bayesian predictive methods for model selection
• The Horseshoe Estimator: Posterior Concentration around Nearly Black Vectors
• Model Selection in Bayesian Neural Networks via Horseshoe Priors
• On the Hyperprior Choice for the Global Shrinkage Parameter in the Horseshoe Prior
• Projection predictive variable selection using Stan+R

Cited by in corpus (54)

• Bayesian Item Response Modeling in R with brms and Stan
• Projective Inference in High-dimensional Problems: Prediction and Feature Selection
• Implicitly Adaptive Importance Sampling
• Stacking for Non-mixing Bayesian Computations: The Curse and Blessing of Multimodal Posteriors
• Variance prior forms for high-dimensional Bayesian variable selection
• Leave-One-Out Cross-Validation for Bayesian Model Comparison in Large Data
• Practical Hilbert space approximate Bayesian Gaussian processes for probabilistic programming
• Intuitive Joint Priors for Bayesian Linear Multilevel Models: The R2D2M2 prior
• Projection Predictive Inference for Generalized Linear and Additive Multilevel Models
• Pathfinder: Parallel quasi-Newton variational inference
• Some models are useful, but how do we know which ones? Towards a unified Bayesian model taxonomy
• Horseshoe Prior Bayesian Quantile Regression
• A fully Bayesian sparse polynomial chaos expansion approach with joint priors on the coefficients and global selection of terms
• Fully Bayesian Classification with Heavy-tailed Priors for Selection in High-dimensional Features with Grouping Structure
• Enforcing stationarity through the prior in vector autoregressions
• The Reciprocal Bayesian LASSO
• Robust Inference and Model Criticism Using Bagged Posteriors
• Prior-preconditioned conjugate gradient method for accelerated Gibbs sampling in "large $n$ & large $p$" Bayesian sparse regression
• Quantifying sources of uncertainty in drug discovery predictions with probabilistic models
• GOES GLM, Biased Bolides, and Debiased Distributions
• Hamiltonian Monte Carlo using an adjoint-differentiated Laplace approximation: Bayesian inference for latent Gaussian models and beyond
• Efficient estimation and correction of selection-induced bias with order statistics
• Adaptive Uncertainty-Guided Model Selection for Data-Driven PDE Discovery
• Inconsistency identification in network meta-analysis via stochastic search variable selection
• Bayesian Inference over the Stiefel Manifold via the Givens Representation
• Latent space projection predictive inference
• Proximal MCMC for Bayesian Inference of Constrained and Regularized Estimation
• Bayesian sparsification for deep neural networks with Bayesian model reduction
• Empirical Bayes inference in sparse high-dimensional generalized linear models
• $R^{\ast }$: A robust MCMC convergence diagnostic with uncertainty using decision tree classifiers
• Informative Bayesian Neural Network Priors for Weak Signals
• Probabilistically-autoencoded horseshoe-disentangled multidomain item-response theory models
• Translating predictive distributions into informative priors
• Projection predictive variable selection for discrete response families with finite support
• Bayesian Joint Modeling of Multiple Brain Functional Networks
• Bayesian Dynamical System Identification With Unified Sparsity Priors And Model Uncertainty
• Adaptive Path Sampling in Metastable Posterior Distributions
• A Laplace Mixture Representation of the Horseshoe and Some Implications
• Sparse encoding for more-interpretable feature-selecting representations in probabilistic matrix factorization
• Pseudo-extended Markov chain Monte Carlo
• Generalized Decomposition Priors on R2
• Design and Structure Dependent Priors for Scale Parameters in Latent Gaussian Models
• Empirical Bayesian Inference using Joint Sparsity
• A fast asynchronous MCMC sampler for sparse Bayesian inference
• Time Fused Coefficient SIR Model with Application to COVID-19 Epidemic in the United States
• Mapping poverty at multiple geographical scales
• Bayesian Regularization: From Tikhonov to Horseshoe
• Strategic Bayesian Asset Allocation
• Copula-like Variational Inference
• HALO: Learning to Prune Neural Networks with Shrinkage
• Horseshoe Forests for High-Dimensional Causal Survival Analysis
• Encoding Domain Information with Sparse Priors for Inferring Explainable Latent Variables
• The ARR2 prior: flexible predictive prior definition for Bayesian auto-regressions
• Gibbs Sampling using Anti-correlation Gaussian Data Augmentation, with Applications to L1-ball-type Models