86 citations · 129 across the 13 of their papers we have counts for
14 papers
Benchmarking and Improving Monitors for Out-Of-Distribution Alignment Failure in LLMs
Dylan Feng, Pragya Srivastava, Anca Dragan +1
Many safety and alignment failures of large language models (LLMs) occur due to out-of-distribution (OOD) situations: unusual prompt or response patterns that are unforeseen by mod…
AssistanceZero: Scalably Solving Assistance Games
Cassidy Laidlaw, Eli Bronstein, Timothy Guo +5
Assistance games are a promising alternative to reinforcement learning from human feedback (RLHF) for training AI assistants. Assistance games resolve key drawbacks of RLHF, such a…
Iterative Label Refinement Matters More than Preference Optimization under Weak Supervision
Yaowen Ye, Cassidy Laidlaw, Jacob Steinhardt
Language model (LM) post-training relies on two stages of human supervision: task demonstrations for supervised finetuning (SFT), followed by preference comparisons for reinforceme…
Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking
Cassidy Laidlaw, Shivam Singhal, Anca Dragan
Because it is difficult to precisely specify complex objectives, reinforcement learning policies are often optimized using proxy reward functions that only approximate the true goa…
Toward Computationally Efficient Inverse Reinforcement Learning via Reward Shaping
Lauren H. Cooke, Harvey Klyne, Edwin Zhang +3
Inverse reinforcement learning (IRL) is computationally challenging, with common approaches requiring the solution of multiple reinforcement learning (RL) sub-problems. This work m…
The Effective Horizon Explains Deep RL Performance in Stochastic Environments
Cassidy Laidlaw, Banghua Zhu, Stuart Russell +1
Reinforcement learning (RL) theory has largely focused on proving minimax sample complexity bounds. These require strategic exploration algorithms that use relatively limited funct…
Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF
Anand Siththaranjan, Cassidy Laidlaw, Dylan Hadfield-Menell
In practice, preference learning from human feedback depends on incomplete data with hidden context. Hidden context refers to data that affects the feedback received, but which is…
Bridging RL Theory and Practice with the Effective Horizon
Cassidy Laidlaw, Stuart Russell, Anca Dragan
Deep reinforcement learning (RL) works impressively in some environments and fails catastrophically in others. Ideally, RL theory should be able to provide an understanding of why…
The Boltzmann Policy Distribution: Accounting for Systematic Suboptimality in Human Models
Cassidy Laidlaw, Anca Dragan
Models of human behavior for prediction and collaboration tend to fall into two categories: ones that learn from large amounts of data via imitation learning, and ones that assume…
Uncertain Decisions Facilitate Better Preference Learning
Cassidy Laidlaw, Stuart Russell
Existing observational approaches for learning human preferences, such as inverse reinforcement learning, usually make strong assumptions about the observability of the human's env…
Perceptual Adversarial Robustness: Defense Against Unseen Threat Models
Cassidy Laidlaw, Sahil Singla, Soheil Feizi
A key challenge in adversarial robustness is the lack of a precise mathematical characterization of human perception, used in the very definition of adversarial attacks that are im…
Playing it Safe: Adversarial Robustness with an Abstain Option
Cassidy Laidlaw, Soheil Feizi
We explore adversarial robustness in the setting in which it is acceptable for a classifier to abstain---that is, output no class---on adversarial examples. Adversarial examples ar…
Functional Adversarial Attacks
Cassidy Laidlaw, Soheil Feizi
We propose functional adversarial attacks, a novel class of threat models for crafting adversarial examples to fool machine learning models. Unlike a standard -ball threat…
Capture, Learning, and Synthesis of 3D Speaking Styles
Daniel Cudeiro, Timo Bolkart, Cassidy Laidlaw +2
Audio-driven 3D facial animation has been widely explored, but achieving realistic, human-like performance is still unsolved. This is due to the lack of available 3D datasets, mode…