Emergent Representations of Program Semantics in Language Models Trained on Programs

Charles Jin, Martin Rinard

We present evidence that language models (LMs) of code can learn to represent the formal semantics of programs, despite being trained only to perform next-token prediction. Specifi…