Solomonoff Induction and Singular Integrals

Abstract

The Solomonoff distribution $M$ assigns an a priori probability to a finite binary string $z$ by summing over all programs whose output begins with $z$, weighting a program of length $\ell$ by $2^{-\ell }$. Thus, likely strings are those with many short explanations. This appears to be a purely discrete notion of complexity. However, Riemann sums are also countable. We show that the sum defining $M$, suitably reorganised, contains Riemann sums approximating the Bayesian evidence of any computable statistical model. These evidence integrals are singular integrals whose asymptotics are governed by Singular Learning Theory and, through it, by invariants of algebraic geometry. Concretely, for every computable Bayesian model, we construct a single monotone Turing machine whose induced semimeasure agrees with the evidence $Z_n$ up to a uniform multiplicative constant. If the model also satisfies the hypotheses of Watanabe's free-energy asymptotics, then a sample $X^n=X_1\cdots X_n$ drawn i.i.d. from the true distribution satisfies $-\log M(X^n)\le n{L}_n(w_0)+\lambda \log n-(m-1)\log \log n+O_{\mathbb{P}}(1)$, where $L_n$ is the empirical loss, $w_0$ is an optimal parameter, $\lambda$ is the learning coefficient, and $m$ is its multiplicity. Thus the learning coefficient, a geometric measure of model simplicity, appears within the Solomonoff distribution as the coefficient of $\log n$ in an upper bound on code length.

19 pages, 2 figures