Variational Gaussian Approximation in Replica Analysis of Parametric Models

Abstract

We revisit the replica method for analyzing inference and learning in parametric models, considering situations where the data-generating distribution is unknown or analytically intractable. Instead of assuming idealized distributions to carry out quenched averages analytically, we use a variational Gaussian approximation for the replicated system in grand canonical formalism in which the data average can be deferred and replaced by empirical averages, leading to stationarity conditions that adaptively determine the parameters of the trial Hamiltonian for each dataset. This approach clarifies how fluctuations affect information extraction and connects directly with the results of mathematical statistics or learning theory such as information criteria. As a concrete application, we analyze linear regression and derive learning curves. This includes cases with real-world datasets, where exact replica calculations are not feasible.

arXiv comment27 pages, 3 figures

References in corpus (10)

• A General Framework for Updating Belief Distributions
• Tractable approximations for probabilistic models: The adaptive TAP mean field approach
• Disorder-free spin glass transitions and jamming in exactly solvable mean-field models
• A statistical mechanics approach to de-biasing and uncertainty estimation in LASSO for random measurements
• From complex to simple : hierarchical free-energy landscape renormalized in deep neural networks
• Statistical Mechanics of Learning: A Variational Approach for Real Data
• Compressed sensing radar detectors under the row-orthogonal design model: a statistical mechanics perspective
• Spatially heterogeneous learning by a deep student machine
• Role of Bootstrap Averaging in Generalized Approximate Message Passing
• Replica Analysis for Ensemble Techniques in Variable Selection