Predictive Statistics Shape Emergent World Representations of Grid Walkers

Sasha Brenner, Thomas R. Knösche, Nico Scherf

Next-token predictors often appear to develop internal representations of the latent world and its rules. The probabilistic nature of these models suggests a deep connection betwee…