5 papers
One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models
Daniel Fein, Max Lamparth, Violet Xiang +2
Reward Models (RMs) are crucial for online alignment of language models (LMs) with human preferences. However, RM-based preference-tuning is vulnerable to reward hacking, whereby L…
Reward Bias Substitution: Single-Axis Bias Mitigations Redirect Optimization Pressure
Max Lamparth, Daniel Fein, Andreas Haupt +2
Single-axis mitigations of reward-model biases (e.g., reducing proxy reliance on length, sycophancy, or style) can rotate optimization pressure onto correlated proxies rather than…
LouvreSAE: Sparse Autoencoders for Interpretable and Controllable Style Transfer
Raina Panda, Daniel Fein, Arpita Singhal +3
Artistic style transfer in generative models remains a significant challenge, as existing methods often introduce style only via model fine-tuning, additional adapters, or prompt e…
Influence Functions for Preference Dataset Pruning
Daniel Fein, Gabriela Aranguiz-Dias
Language models are commonly fine-tuned via reinforcement learning to alter their behavior or elicit new capabilities. Datasets used for these purposes, and particularly human pref…
LitBench: A Benchmark and Dataset for Reliable Evaluation of Creative Writing
Daniel Fein, Sebastian Russo, Violet Xiang +3
Evaluating creative writing generated by large language models (LLMs) remains challenging because open-ended narratives lack ground truths. Without performant automated evaluation…