4 papers
Learning State-Tracking from Code Using Linear RNNs
Julien Siems, Riccardo Grazzi, Korbinian Pöppel +3
Over the last years, state-tracking tasks, particularly permutation composition, have become a testbed to understand the limits of sequence models architectures like Transformers a…
RevengeBench: Reverse Engineering Code-Space Policies from Behavioral Experiments
Babak Rahmani, Sebastian Dziadzio, Joschka Strüber +2
For most of scientific history, researchers studying behavior could only infer hidden mechanisms from outward actions: an inverse problem that becomes more tractable when observati…
Debugging code world models
Babak Rahmani
Code World Models (CWMs) are language models trained to simulate program execution by predicting explicit runtime state after every executed command. This execution-based world mod…
Implicit Language Models are RNNs: Balancing Parallelization and Expressivity
Mark Schöne, Babak Rahmani, Heiner Kremer +3
State-space models (SSMs) and transformers dominate the language modeling landscape. However, they are constrained to a lower computational complexity than classical recurrent neur…