Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory

Abstract

In regular statistical models, the leave-one-out cross-validation is asymptotically equivalent to the Akaike information criterion. However, since many learning machines are singular statistical models, the asymptotic behavior of the cross-validation remains unknown. In previous studies, we established the singular learning theory and proposed a widely applicable information criterion, the expectation value of which is asymptotically equal to the average Bayes generalization loss. In the present paper, we theoretically compare the Bayes cross-validation loss and the widely applicable information criterion and prove two theorems. First, the Bayes cross-validation loss is asymptotically equivalent to the widely applicable information criterion as a random variable. Therefore, model selection and hyperparameter optimization using these two values are asymptotically equivalent. Second, the sum of the Bayes generalization error and the Bayes cross-validation error is asymptotically equal to $2\lambda /n$, where $\lambda$ is the real log canonical threshold and $n$ is the number of training samples. Therefore the relation between the cross-validation error and the generalization error is determined by the algebraic geometrical structure of a learning machine. We also clarify that the deviance information criteria are different from the Bayes cross-validation and the widely applicable information criterion.

References in corpus (4)

• On real log canonical thresholds
• Asymptotic Learning Curve and Renormalizable Condition in Statistical Learning Theory
• Equations of States in Statistical Learning for a Nonparametrizable and Regular Case
• An asymptotic approximation of the marginal likelihood for general Markov models

Cited by in corpus (167)

• Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
• The Electromagnetic Counterpart of the Binary Neutron Star Merger LIGO/VIRGO GW170817. II. UV, Optical, and Near-IR Light Curves and Comparison to Kilonova Models
• A Widely Applicable Bayesian Information Criterion
• Using stacking to average Bayesian predictive distributions
• Comparison of Bayesian predictive methods for model selection
• Statistical methods for linguistic research: Foundational Ideas - Part II
• The magnetar model for Type I superluminous supernovae I: Bayesian analysis of the full multicolour light curve sample with MOSFiT
• True Few-Shot Learning with Language Models
• WAIC, but Why? Generative Ensembles for Robust Anomaly Detection
• Choice modelling in the age of machine learning -- discussion paper
• A thousand days after the merger: continued X-ray emission from GW170817
• Bayesian comparison of latent variable models: Conditional vs marginal likelihoods
• Efficient Bayesian Structural Equation Modeling in Stan
• Statistical methods for linguistic research: Foundational Ideas - Part I
• Bayesian latent time joint mixed effect models for multicohort longitudinal data
• A multi-messenger study of the Milky Way's stellar disc and bulge with LISA, Gaia and LSST
• A Bayesian Cognition Approach to Improve Data Visualization
• Automatic cross-validation in structured models: Is it time to leave out leave-one-out?
• Bayesian inference and model comparison for metallic fatigue data
• A resonant sextuplet of sub-Neptunes transiting the bright star HD 110067
• Maximizing the information learned from finite data selects a simple model
• An extremely energetic supernova from a very massive star in a dense medium
• Bayesian leave-one-out cross-validation approximations for Gaussian latent variable models
• Affinity and Diversity: Quantifying Mechanisms of Data Augmentation
• Testing dark matter and geometry sustained circular velocities in the Milky Way with Gaia DR2
• Unbiased and Efficient Log-Likelihood Estimation with Inverse Binomial Sampling
• A tale of two mergers: constraints on kilonova detection in two short GRBs at z$\sim$0.5
• Astrometry and exoplanets in the Gaia era: a Bayesian approach to detection and parameter recovery
• Informed attribution of flood changes to decadal variation of atmospheric, catchment and river drivers in Upper Austria
• bamlss: A Lego Toolbox for Flexible Bayesian Regression (and Beyond)
• Bayesian Modeling with Gaussian Processes using the GPstuff Toolbox
• Approximating Cross-validatory Predictive Evaluation in Bayesian Latent Variables Models with Integrated IS and WAIC
• Leave-One-Out Cross-Validation for Bayesian Model Comparison in Large Data
• Applying Bayesian Analysis Guidelines to Empirical Software Engineering Data: The Case of Programming Languages and Code Quality
• The role of alcohol outlet visits derived from mobile phone location data in enhancing domestic violence prediction at the neighborhood level
• Parameter inference and model selection in deterministic and stochastic dynamical models via approximate Bayesian computation: modeling a wildlife epidemic
• spAbundance: An R package for single-species and multi-species spatially explicit abundance models
• GPdoemd: a Python package for design of experiments for model discrimination
• Guidelines for the use of spatially-varying coefficients in species distribution models
• Resolved stellar streams around NGC4631 from a Subaru/Hyper Suprime-Cam survey
• Robust leave-one-out cross-validation for high-dimensional Bayesian models
• Spatial Power Spectra of Dust across the Local Group: No Constraint on Disc Scale Height
• Predictive Bayesian selection of multistep Markov chains, applied to the detection of the hot hand and other statistical dependencies in free throws
• Multivariate hierarchical analysis of car crashes data considering a spatial network lattice
• Latent Gaussian modeling and INLA: A review with focus on space-time applications
• Big problems in spatio-temporal disease mapping: methods and software
• Renormalisation of linear halo bias in N-body simulations
• Some models are useful, but how do we know which ones? Towards a unified Bayesian model taxonomy
• Bayesian Variational Autoencoders for Unsupervised Out-of-Distribution Detection
• PriorVAE: Encoding spatial priors with VAEs for small-area estimation
• Bayesian Non-Parametric Detection Heterogeneity in Ecological Models
• Bayesian Inference of Local Projections with Roughness Penalty Priors
• Scalable Bayesian Approach for the DINA Q-matrix Estimation Combining Stochastic Optimization and Variational Inference
• Information Theory and Statistics: an overview
• Bayesian Modeling of Dynamic Behavioral Change During an Epidemic
• Improving parametric mass modelling of lensing clusters through a perturbative approach
• Modeling the effects of telephone nursing on healthcare utilization
• Enhanced monitoring of atmospheric methane from space over the Permian basin with hierarchical Bayesian inference
• Heterogeneous Distributed Lag Models to Estimate Personalized Effects of Maternal Exposures to Air Pollution
• Bayesian models for prediction of the set-difference in volleyball
• A Note on Bayesian Modeling Specification of Censored Data in JAGS
• Holdout predictive checks for Bayesian model criticism
• High-dimensional order-free multivariate spatial disease mapping
• Fast and accurate Bayesian model criticism and conflict diagnostics using R-INLA
• Modeling Extremal Streamflow using Deep Learning Approximations and a Flexible Spatial Process
• Bayesian Model Selection via Mean-Field Variational Approximation
• Integrated Nested Laplace Approximations (INLA)
• Understanding predictive information criteria for Bayesian models
• Variational Approximation Error in Bayesian Non-negative Matrix Factorization
• Dimension-Grouped Mixed Membership Models for Multivariate Categorical Data
• Bayesian Information Criterion for Linear Mixed-effects Models
• A Bayesian Approach to Modelling Fine-Scale Spatial Dynamics of Non-State Terrorism: World Study, 2002-2013
• Adaptive Uncertainty-Guided Model Selection for Data-Driven PDE Discovery
• A Review of Statistical Methods in Imaging Genetics
• Bayesian Modal Regression based on Mixture Distributions
• Bayesian item response models for citizen science ecological data
• Income, education, and other poverty-related variables: a journey through Bayesian hierarchical models
• Bayesian Computing with INLA: A Review
• A Bayesian approach to uncover local and temporal determinants of heterogeneity in repeated cross-sectional health surveys
• How to select an objective function using information theory
• Probabilistically-autoencoded horseshoe-disentangled multidomain item-response theory models
• Measurement Error Models for Spatial Network Lattice Data: Analysis of Car Crashes in Leeds
• varstan: An R package for Bayesian analysis of structured time series models with Stan
• Warped Dynamic Linear Models for Time Series of Counts
• Bayesian inference for dynamic vine copulas in higher dimensions
• T-LoHo: A Bayesian Regularization Model for Structured Sparsity and Smoothness on Graphs
• What are the most important statistical ideas of the past 50 years?
• Bayesian Cross Validation and WAIC for Predictive Prior Design in Regular Asymptotic Theory
• Spatial Modeling for Correlated Cancers Using Bivariate Directed Graphs
• Personalized Dynamic Treatment Regimes in Continuous Time: A Bayesian Approach for Optimizing Clinical Decisions with Timing
• Stochastic Volatily Models using Hamiltonian Monte Carlo Methods and Stan
• Information criteria for non-normalized models
• Statistical Learning Theory of Quasi-Regular Cases
• Horseshoe-based Bayesian nonparametric estimation of effective population size trajectories
• A scalable approach for short-term disease forecasting in high spatial resolution areal data
• Accounting for spatial varying sampling effort due to accessibility in Citizen Science data: A case study of moose in Norway
• Estimating Real Log Canonical Thresholds
• An asymptotic approximation of the marginal likelihood for general Markov models
• A Novel Class of Unfolding Models for Binary Preference Data
• Divergent Effects of Factors on Crashes under Autonomous and Conventional Driving Modes Using A Hierarchical Bayesian Approach
• Adaptive preferential sampling in phylodynamics
• Bigeminal Priors Variational auto-encoder
• A Two-Stage Cox Process Model with Spatial and Nonspatial Covariates
• A priori truncation method for posterior sampling from homogeneous normalized completely random measure mixture models
• Bayesian Mendelian Randomization
• Approximate Bayesian Inference via Sparse grid Quadrature Evaluation for Hierarchical Models
• Bayesian hierarchical space-time models to improve multispecies assessment by combining observations from disparate fish surveys
• A Gentle Introduction to Bayesian Hierarchical Linear Regression Models
• On prior smoothing with discrete spatial data in the context of disease mapping
• Estimating Heterogeneous Exposure Effects in the Case-Crossover Design using BART
• Sparse encoding for more-interpretable feature-selecting representations in probabilistic matrix factorization
• Statistical modeling of groundwater quality assessment in Iran using a flexible Poisson likelihood
• A Latent Space Model for Cognitive Social Structures Data
• Regularization Parameter Selection for a Bayesian Multi-Level Group Lasso Regression Model with Application to Imaging Genomics
• Gaussian Process Single Index Models for Conditional Copulas
• Modeling and presentation of vaccination coverage estimates using data from household surveys
• Fast Bayesian inference of Block Nearest Neighbor Gaussian process for large data
• The Exact Asymptotic Form of Bayesian Generalization Error in Latent Dirichlet Allocation
• Regularized Bayesian calibration and scoring of the WD-FAB IRT model improves predictive performance over marginal maximum likelihood
• Large Deviation Strategy for Inverse Problem
• Parameter Recovery with Marginal Maximum Likelihood and Markov Chain Monte Carlo Estimation for the Generalized Partial Credit Model
• A Record Linkage Model Incorporating Relational Data
• Prediction can be safely used as a proxy for explanation in causally consistent Bayesian generalized linear models
• A scale of interpretation for likelihood ratios and Bayes factors
• Diagnostic-Driven Nonstationary Emulators Using Kernel Mixtures
• Empirical Bayes factors for common hypothesis tests
• CopulaDTA: An R Package for Copula Based Bivariate Beta-Binomial Models for Diagnostic Test Accuracy Studies in a Bayesian Framework
• Bayesian Measurement Error Correction in Structured Additive Distributional Regression with an Application to the Analysis of Sensor Data on Soil-Plant Variability
• Adaptive shrinkage of smooth functional effects towards a predefined functional subspace
• Hierarchical Multivariate Directed Acyclic Graph Auto-Regressive (MDAGAR) models for spatial diseases mapping
• A Latent Space Model for Multilayer Network Data
• Non-Separable Spatio-temporal Models via Transformed Gaussian Markov Random Fields
• Simultaneous estimation of the effective reproducing number and the detection rate of COVID-19
• Bayesian Hierarchical Bernoulli-Weibull Mixture Model for Extremely Rare Events
• Dynamic Factor Models for Binary Data in Circular Spaces: An Application to the U.S. Supreme Court
• Joint Quantile Regression for Spatial Data
• Bayesian Regression Markets
• An Embedded Diachronic Sense Change Model with a Case Study from Ancient Greek
• CALF-SBM: A Covariate-Assisted Latent Factor Stochastic Block Model
• BDSAR: a new package on Bregman divergence for Bayesian simultaneous autoregressive models
• A spatio-temporal process-convolution model for quantifying health inequalities in respiratory prescription rates in Scotland
• Addressing outliers in mixed-effects logistic regression: a more robust modeling approach
• Significant cocoon emission and photosphere duration stretching in GRB 211211A: a burst from a neutron star - black hole merger
• Semi-Supervised Few-Shot Classification with Deep Invertible Hybrid Models
• Spatial modelling of COVID-19 incident cases using Richards' curve: an application to the Italian regions
• Posterior Dispersion Indices
• Joint Dispersion Model with a Flexible Link
• Relevance Vector Machine with Weakly Informative Hyperprior and Extended Predictive Information Criterion
• A numerically stable online implementation and exploration of WAIC through variations of the predictive density, using NIMBLE
• A Partially Non-Ergodic Ground-Motion Prediction Equation for Europe
• Bayesian factor models for multivariate categorical data obtained from questionnaires
• The Quantum Relative Entropy as a Rate Function and Information Criteria
• Flexible Validity Conditions for the Multivariate Matérn Covariance in any Spatial Dimension and for any Number of Components
• Asymptotic Analysis of the Bayesian Likelihood Ratio for Testing Homogeneity in Normal Mixture Models
• Bayesian Model Selection for a Class of Spatially-Explicit Capture Recapture Models
• Bivariate Analysis of Birth Weight and Gestational Age Depending on Environmental Exposures: Bayesian Distributional Regression with Copulas
• Semiparametric Functional Factor Models with Bayesian Rank Selection
• Bayesian modelling of statistical region- and family-level clustered ordinal outcome data from Turkey
• Testing for Typicality with Respect to an Ensemble of Learned Distributions
• Entropic Issues in Likelihood-Based OOD Detection
• Analysis of Prescription Drug Utilization with Beta Regression Models
• An Introduction to Inductive Statistical Inference: from Parameter Estimation to Decision-Making
• Prior Intensified Information Criterion
• Bayesian model selection in the $\mathcal{M}$-open setting -- Approximate posterior inference and probability-proportional-to-size subsampling for efficient large-scale leave-one-out cross-validation
• Bayesian Estimation of Multidimensional Latent Variables and Its Asymptotic Accuracy
• Competition among eggs shifts to cooperation along a sperm supply gradient in an external fertilizer
• A spatio-temporal model to understand forest fires causality in Europe