6 papers
Forking Fast: Efficiently Estimating Uncertainty Dynamics in Text Generation
Eric Bigelow, Amir Zur, Satchel Grant +7
LLM reasoning is stochastic, and so understanding a model requires grappling with the distribution of reasoning chains that it might produce for a given question, i.e., its uncerta…
Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signal
Leon Bergen, Usha Bhalla, Sidharth Baskaran +14
Language-model post-training is the main stage at which model behavior is shaped, yet it still largely involves optimization of scalar rewards that summarize diverse desiderata. Th…
Stories in Space: In-Context Learning Trajectories in Conceptual Belief Space
Eric Bigelow, Raphaël Sarfati, Daniel Wurgaft +5
Large Language Models (LLMs) update their behavior in context, which can be viewed as a form of Bayesian inference. However, the structure of the latent hypothesis space over which…
Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior
Daniel Wurgaft, Can Rager, Matthew Kowal +13
Neural representations carry rich geometric structure; but does that structure causally shape behavior? To address this question, we intervene along paths through activation space…
Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Concepts
Sheridan Feucht, Tal Haklay, Usha Bhalla +9
Does structure in representations imply structure in computation? We study how Llama-3.1-8B reasons over cyclic concepts (e.g., "what month is six months after August?"). Even thou…
Shifting the Gradient: Understanding How Defensive Training Methods Protect Language Model Integrity
Satchel Grant, Victor Gillioz, Jake Ward +1
Defensive training methods such as positive preventative steering (PPS) and inoculation prompting (IP) offer surprising results through seemingly similar processes: both add trait-…