97 citations · 109 across the 14 of their papers we have counts for
14 papers
Model Spec Midtraining: Improving How Alignment Training Generalizes
Chloe Li, Nevan Wichers, Sara Price +2
Some frontier AI developers aim to align language models to a Model Spec or Constitution that describes the intended model behavior. However, standard alignment fine-tuning -- trai…
Recontextualization Mitigates Specification Gaming without Modifying the Specification
Ariana Azarbal, Victor Gillioz, Vladimir Ivanov +6
Developers often struggle to specify correct training labels and rewards. Perhaps they don't need to. We propose recontextualization, which reduces how often language models "game"…
Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignment
Nevan Wichers, Aram Ebtekar, Ariana Azarbal +8
Large language models are sometimes trained with imperfect oversight signals, leading to undesired behaviors such as reward hacking and sycophancy. Improving oversight quality can…
Visualizing Neural Network Imagination
Nevan Wichers, Victor Tao, Riccardo Volpato +1
In certain situations, neural networks will represent environment states in their hidden activations. Our goal is to visualize what environment states the networks are representing…
Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generation
Meng Cao, Lei Shu, Lei Yu +4
Reinforcement learning (RL) can align language models with non-differentiable reward signals, such as human preferences. However, a major challenge arises from the sparsity of thes…
Gradient-Based Language Model Red Teaming
Nevan Wichers, Carson Denison, Ahmad Beirami
Red teaming is a common strategy for identifying weaknesses in generative language models (LMs), where adversarial prompts are produced that trigger an LM to generate unsafe respon…