Transformers As Approximations of Solomonoff Induction

Nathan Young, Michael Witbrock

Solomonoff Induction is an optimal-in-the-limit unbounded algorithm for sequence prediction, representing a Bayesian mixture of every computable probability distribution and perfor…