Dynamics of Bayesian Updating with Dependent Data and Misspecified Models

Abstract

Much is now known about the consistency of Bayesian updating on infinite-dimensional parameter spaces with independent or Markovian data. Necessary conditions for consistency include the prior putting enough weight on the correct neighborhoods of the data-generating distribution; various sufficient conditions further restrict the prior in ways analogous to capacity control in frequentist nonparametrics. The asymptotics of Bayesian updating with mis-specified models or priors, or non-Markovian data, are far less well explored. Here I establish sufficient conditions for posterior convergence when all hypotheses are wrong, and the data have complex dependencies. The main dynamical assumption is the asymptotic equipartition (Shannon-McMillan-Breiman) property of information theory. This, along with Egorov's Theorem on uniform convergence, lets me build a sieve-like structure for the prior. The main statistical assumption, also a form of capacity control, concerns the compatibility of the prior and the data-generating process, controlling the fluctuations in the log-likelihood when averaged over the sieve-like sets. In addition to posterior convergence, I derive a kind of large deviations principle for the posterior measure, extending in some cases to rates of convergence, and discuss the advantages of predicting using a combination of models known to be wrong. An appendix sketches connections between these results and the replicator dynamics of evolutionary theory.

36 pages, 1 figure. v2: typo fixes, minor formatting changes. v3: Improved notation, added references, new theorem on convergence rates. v4: minor changes to text, added references. v5: Minor typo corrections; matches journal version except for format details

References in corpus (8)

• Convergence rates of posterior distributions for noniid observations
• From $\epsilon$-entropy to KL-entropy: Analysis of minimum information complexity density estimation
• Misspecification in infinite-dimensional Bayesian statistics
• New approaches to Bayesian consistency
• Coupled Replicator Equations for the Dynamics of Learning in Multiagent Systems
• Inferring Markov Chains: Bayesian Estimation, Model Comparison, Entropy Rate, and Out-of-class Modeling
• Remarks on consistency of posterior distributions
• From Finite to Infinite Range Order via Annealing: The Causal Architecture of Deformation Faulting in Annealed Close-Packed Crystals

Cited by in corpus (15)

• Philosophy and the practice of Bayesian statistics
• Thermodynamics as a theory of decision-making with information processing costs
• Universal Darwinism as a process of Bayesian inference
• On the Brittleness of Bayesian Inference
• The Price equation program: simple invariances unify population dynamics, thermodynamics, probability, information and inference
• Sequential anomaly detection in the presence of noise and limited feedback
• On Posterior Concentration in Misspecified Models
• Universal expressions of population change by the Price equation: natural selection, information, and maximum entropy production
• Bayesian model comparison with the Hyvärinen score: computation and consistency
• Natural selection. V. How to read the fundamental equations of evolutionary change in terms of information theory
• Simple unity among the fundamental equations of science
• Consistency of maximum likelihood estimation for some dynamical systems
• Stationary Stability for Evolutionary Dynamics in Finite Populations
• The Price equation reveals a universal force-metric-bias law of algorithmic learning and natural selection
• Discerning media bias within a network of political allies: an analytic condition for disruption by partisans