A Causal World Model Underlying Next Token Prediction: Exploring GPT in a Controlled Environment

Raanan Y. Rohekar, Yaniv Gurwicz, Sungduk Yu +2

Are generative pre-trained transformer (GPT) models, trained only to predict the next token, implicitly learning a world model from which sequences are generated one token at a tim…