Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task

Abstract

Language models show a surprising range of capabilities, but the source of their apparent competence is unclear. Do these networks just memorize a collection of surface statistics, or do they rely on internal representations of the process that generates the sequences they see? We investigate this question by applying a variant of the GPT model to the task of predicting legal moves in a simple board game, Othello. Although the network has no a priori knowledge of the game or its rules, we uncover evidence of an emergent nonlinear internal representation of the board state. Interventional experiments indicate this representation can be used to control the output of the network and create "latent saliency maps" that can help explain predictions in human terms.

ICLR 2023 oral (notable-top-5%): https://openreview.net/forum?id=DeG07_TcZvT ; code: https://github.com/likenneth/othello_world

Cited by in corpus (8)

• From task structures to world models: What do LLMs know?
• Domain-specific ChatBots for Science using Embeddings
• Opportunities for Large Language Models and Discourse in Engineering Design
• Standards for Belief Representations in LLMs
• What makes a language easy to deep-learn? Deep neural networks and humans similarly benefit from compositional structure
• Chatting with Bots: AI, Speech Acts, and the Edge of Assertion
• cosmosage: A Natural-Language Assistant for Cosmologists
• Divergences between Language Models and Human Brains