97 citations · 109 across the 14 of their papers we have counts for
14 papers
Model Spec Midtraining: Improving How Alignment Training Generalizes
Chloe Li, Nevan Wichers, Sara Price +2
Some frontier AI developers aim to align language models to a Model Spec or Constitution that describes the intended model behavior. However, standard alignment fine-tuning -- trai…
Recontextualization Mitigates Specification Gaming without Modifying the Specification
Ariana Azarbal, Victor Gillioz, Vladimir Ivanov +6
Developers often struggle to specify correct training labels and rewards. Perhaps they don't need to. We propose recontextualization, which reduces how often language models "game"…
Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignment
Nevan Wichers, Aram Ebtekar, Ariana Azarbal +8
Large language models are sometimes trained with imperfect oversight signals, leading to undesired behaviors such as reward hacking and sycophancy. Improving oversight quality can…
Visualizing Neural Network Imagination
Nevan Wichers, Victor Tao, Riccardo Volpato +1
In certain situations, neural networks will represent environment states in their hidden activations. Our goal is to visualize what environment states the networks are representing…
Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generation
Meng Cao, Lei Shu, Lei Yu +4
Reinforcement learning (RL) can align language models with non-differentiable reward signals, such as human preferences. However, a major challenge arises from the sparsity of thes…
Gradient-Based Language Model Red Teaming
Nevan Wichers, Carson Denison, Ahmad Beirami
Red teaming is a common strategy for identifying weaknesses in generative language models (LMs), where adversarial prompts are produced that trigger an LM to generate unsafe respon…
SiRA: Sparse Mixture of Low Rank Adaptation
Yun Zhu, Nevan Wichers, Chu-Cheng Lin +8
Parameter Efficient Tuning has been an prominent approach to adapt the Large Language Model to downstream tasks. Most previous works considers adding the dense trainable parameters…
Fusion-Eval: Integrating Assistant Evaluators with LLMs
Lei Shu, Nevan Wichers, Liangchen Luo +4
Evaluating natural language systems poses significant challenges, particularly in the realms of natural language understanding and high-level reasoning. In this paper, we introduce…
SAFER: Data-Efficient and Safe Reinforcement Learning via Skill Acquisition
Dylan Slack, Yinlam Chow, Bo Dai +1
Methods that extract policy primitives from offline demonstrations using deep generative models have shown promise at accelerating reinforcement learning(RL) for new tasks. Intuiti…
ActionBert: Leveraging User Actions for Semantic Understanding of User Interfaces
Zecheng He, Srinivas Sunkara, Xiaoxue Zang +7
As mobile devices are becoming ubiquitous, regularly interacting with a variety of user interfaces (UIs) is a common aspect of daily life for many people. To improve the accessibil…
RL agents Implicitly Learning Human Preferences
Nevan Wichers
In the real world, RL agents should be rewarded for fulfilling human preferences. We show that RL agents implicitly learn the preferences of humans in their environment. Training a…
Resolving Spurious Correlations in Causal Models of Environments via Interventions
Sergei Volodin, Nevan Wichers, Jeremy Nixon
Causal models bring many benefits to decision-making systems (or agents) by making them interpretable, sample-efficient, and robust to changes in the input distribution. However, s…
Resolving Referring Expressions in Images With Labeled Elements
Nevan Wichers, Dilek Hakkani-Tur, Jindong Chen
Images may have elements containing text and a bounding box associated with them, for example, text identified via optical character recognition on a computer screen image, or a na…
Hierarchical Long-term Video Prediction without Supervision
Nevan Wichers, Ruben Villegas, Dumitru Erhan +1
Much of recent research has been devoted to video prediction and generation, yet most of the previous works have demonstrated only limited success in generating videos on short-ter…