A Widely Applicable Bayesian Information Criterion

Abstract

A statistical model or a learning machine is called regular if the map taking a parameter to a probability distribution is one-to-one and if its Fisher information matrix is always positive definite. If otherwise, it is called singular. In regular statistical models, the Bayes free energy, which is defined by the minus logarithm of Bayes marginal likelihood, can be asymptotically approximated by the Schwarz Bayes information criterion (BIC), whereas in singular models such approximation does not hold. Recently, it was proved that the Bayes free energy of a singular model is asymptotically given by a generalized formula using a birational invariant, the real log canonical threshold (RLCT), instead of half the number of parameters in BIC. Theoretical values of RLCTs in several statistical models are now being discovered based on algebraic geometrical methodology. However, it has been difficult to estimate the Bayes free energy using only training samples, because an RLCT depends on an unknown true distribution. In the present paper, we define a widely applicable Bayesian information criterion (WBIC) by the average log likelihood function over the posterior distribution with the inverse temperature $1/\log n$, where $n$ is the number of training samples. We mathematically prove that WBIC has the same asymptotic expansion as the Bayes free energy, even if a statistical model is singular for and unrealizable by a statistical model. Since WBIC can be numerically calculated without any information about a true distribution, it is a generalized version of BIC onto singular statistical models.

30 pages

References in corpus (4)

• Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory
• Likelihood ratio tests and singularities
• On real log canonical thresholds
• Asymptotic Learning Curve and Renormalizable Condition in Statistical Learning Theory

Cited by in corpus (62)

• Maximizing the information learned from finite data selects a simple model
• Astrometry and exoplanets in the Gaia era: a Bayesian approach to detection and parameter recovery
• Video-rate multispectral imaging in laparoscopic surgery: First-in-human application
• Malware in the Future? Forecasting of Analyst Detection of Cyber Events
• GPdoemd: a Python package for design of experiments for model discrimination
• Predictive Bayesian selection of multistep Markov chains, applied to the detection of the hot hand and other statistical dependencies in free throws
• Subspace Inference for Bayesian Deep Learning
• Brain tumour genetic network signatures of survival
• Empirical Survival Jensen-Shannon Divergence as a Goodness-of-Fit Measure for Maximum Likelihood Estimation and Curve Fitting
• The Bayesian Asteroseismology data Modeling pipeline and its application to $K2$ data
• Scalable Bayesian Approach for the DINA Q-matrix Estimation Combining Stochastic Optimization and Variational Inference
• A Bayesian semi-parametric approach for modeling memory decay in dynamic social networks
• Health improvement framework for planning actionable treatment process using surrogate Bayesian model
• Bayesian Non-parametric Quantile Process Regression and Estimation of Marginal Quantile Effects
• Modeling of the parties' vote share distributions
• Bayesian Multivariate Spatial Models for Lattice Data with INLA
• BAMBI: An R package for Fitting Bivariate Angular Mixture Models
• A Degradation Performance Model With Mixed-type Covariates and Latent Heterogeneity
• Understanding predictive information criteria for Bayesian models
• A Kernel Stein Test for Comparing Latent Variable Models
• $J$-factor estimation of Draco, Sculptor and Ursa Minor dwarf spheroidal galaxies with the member/foreground mixture model
• Deep Learning is Singular, and That's Good
• A Parsimonious Tour of Bayesian Model Uncertainty
• Rebuilding Factorized Information Criterion: Asymptotically Accurate Marginal Likelihood
• Assessing the Performance of Diagnostic Classification Models in Small Sample Contexts with Different Estimation Methods
• Predicting exoplanet mass from radius and incident flux: A Bayesian mixture model
• Bayesian Generalization Error of Poisson Mixture and Simplex Vandermonde Matrix Type Singularity
• Finite Population Survey Sampling: An Unapologetic Bayesian Perspective
• Informative Bayesian model selection for RR Lyrae star classifiers
• A Novel Class of Unfolding Models for Binary Preference Data
• Do GPT Language Models Suffer From Split Personality Disorder? The Advent Of Substrate-Free Psychometrics
• Estimating Real Log Canonical Thresholds
• Stochastic Volatily Models using Hamiltonian Monte Carlo Methods and Stan
• Statistical Models for Repeated Categorical Ratings: The R Package rater
• Directed Cyclic Graph for Causal Discovery from Multivariate Functional Data
• A Latent Space Model for Cognitive Social Structures Data
• A general framework for modelling zero inflation
• A Gentle Introduction to Bayesian Hierarchical Linear Regression Models
• Regularized Bayesian calibration and scoring of the WD-FAB IRT model improves predictive performance over marginal maximum likelihood
• Bayesian Hierarchical Bernoulli-Weibull Mixture Model for Extremely Rare Events
• A Latent Space Model for Multilayer Network Data
• Geometry of Program Synthesis
• MONCAE: Multi-Objective Neuroevolution of Convolutional Autoencoders
• Region-Referenced Spectral Power Dynamics of EEG Signals: A Hierarchical Modeling Approach
• A Record Linkage Model Incorporating Relational Data
• Phylogeny-based tumor subclone identification using a Bayesian feature allocation model
• Forecasting under model uncertainty:Non-homogeneous hidden Markov models with Polya-Gamma data augmentation
• The Exact Asymptotic Form of Bayesian Generalization Error in Latent Dirichlet Allocation
• Frequency behaviour for multinomial counts of fisheries discards via a nested wavelet zero and N inflated binomial model
• Applying Bayesian Hierarchical Probit Model to Interview Grade Evaluation
• spsurv: An R package for semi-parametric survival analysis
• On the overestimation of widely applicable Bayesian information criterion
• Convergence Rates of Latent Topic Models Under Relaxed Identifiability Conditions
• Analytic solution and stationary phase approximation for the Bayesian lasso and elastic net
• Differential Description Length for Hyperparameter Selection in Machine Learning
• Semi-Parametric Hierarchical Bayes Estimates of New Yorkers' Willingness to Pay for Features of Shared Automated Vehicle Services
• Asyptotic Normality for Maximum Likelihood Estimation and Operational Risk
• Dynamic Factor Models for Binary Data in Circular Spaces: An Application to the U.S. Supreme Court
• Body movement to sound interface with vector autoregressive hierarchical hidden Markov models
• Relevance Vector Machine with Weakly Informative Hyperprior and Extended Predictive Information Criterion
• A flexible Bayesian non-confounding spatial model for analysis of dispersed count data in clinical studies
• Discussion of: "A Bayesian information criterion for singular models"