Intrinsic regularization effect in Bayesian nonlinear regression scaled by observed data

Abstract

Occam's razor is a guiding principle that models should be simple enough to describe observed data. While Bayesian model selection (BMS) embodies it by the intrinsic regularization effect (IRE), how observed data scale the IRE has not been fully understood. In the nonlinear regression with conditionally independent observations, we show that the IRE is scaled by observations' fineness, defined by the amount and quality of observed data. We introduce an observable that quantifies the IRE, referred to as the Bayes specific heat, inspired by the correspondence between statistical inference and statistical physics. We derive its scaling relation to observations' fineness. We demonstrate that the optimal model chosen by the BMS changes at critical values of observations' fineness, accompanying the IRE's variation. The changes are from choosing a coarse-grained model to a fine-grained one as observations' fineness increases. Our findings expand an understanding of BMS's typicality when observed data are insufficient.

References in corpus (13)

• Exchange Monte Carlo Method and Application to Spin Glass Simulations
• Bayes in the sky: Bayesian inference and model selection in cosmology
• Statistical physics of inference: Thresholds and algorithms
• Parameter Space Compression Underlies Emergent Theories and Predictive Models
• The Nishimori line and Bayesian Statistics
• Maximizing the information learned from finite data selects a simple model
• Simultaneous Estimation of Noise Variance and Number of Peaks in Bayesian Spectral Deconvolution
• Non-Markov property of certain eigenvalue processes analogous to Dyson's model
• Bayesian model selection on Scalar $\epsilon$-Field Dark Energy
• On the correspondence between thermodynamics and inference
• Relevance in the Renormalization Group and in Information Theory
• Unveiling quasiparticle dynamics of topological insulators through Bayesian modelling
• Bayesian inference of ion velocity distribution function from laser-induced fluorescence spectra