6 papers
Subliminal Learning is a LoRA Artifact
Todd Nief, Harvey Yiyun Fu, Mark Muchane +1
Subliminal learning is a phenomenon where language models can transmit behavioral traits to other models through seemingly innocuous data (Cloud et al., 2025). In subliminal learni…
The Information Geometry of Softmax: Probing and Steering
Kiho Park, Todd Nief, Yo Joong Choe +1
This paper concerns the question of how AI systems encode semantic structure into the geometric structure of their representation spaces. The motivating observation is that the nat…
Iterative Finetuning is Mostly Idempotent
Zephaniah Roe, Jack Sanderson, Dang Nguyen +5
If a model has some behavioral tendency, such as sycophancy or misalignment, and it is trained on its own outputs, will the tendency be amplified in the next generation of models?…
Dynamic Weight Grafting: Localizing Finetuned Factual Knowledge in Transformers
Todd Nief, David Reber, Sean Richardson +1
When an LLM learns a new fact during finetuning (e.g., new movie releases, newly elected pope, etc.), where does this information go? Are entities enriched with relation informatio…
RATE: Causal Explainability of Reward Models with Imperfect Counterfactuals
David Reber, Sean Richardson, Todd Nief +2
Reward models are widely used as proxies for human preferences when aligning or evaluating LLMs. However, reward models are black boxes, and it is often unclear what, exactly, they…
Deep Model Merging: The Sister of Neural Network Interpretability -- A Survey
Arham Khan, Todd Nief, Nathaniel Hudson +6
We survey the model merging literature through the lens of loss landscape geometry to connect observations from empirical studies on model merging and loss landscape analysis to ph…