Better World Models Can Lead to Better Post-Training Performance

Prakhar Gupta, Henry Conklin, Sarah-Jane Leslie +1

We study how explicit world-modeling objectives affect the internal representations and downstream capability of Transformers, using Rubik's Cubes as our training domain. We ask: (…