Constrained belief updates explain geometric structures in transformer representations

Mateusz Piotrowski, Paul M. Riechers, Daniel Filan +1

What computational structures emerge in transformers trained on next-token prediction? In this work, we provide evidence that transformers implement constrained Bayesian belief upd…