Small transformers track Bayesian evidence for latent common causes via a context-invariant mechanism

Amir Mohammadpour, Michael Franke

We present an in-depth investigation of how a form of Bayesian reasoning about common causes can emerge as a cross-contextual generalization in small, tractable transformers. Incre…