387 citations · 971 across the 40 of their papers we have counts for
14 papers · 1 filter
Distributional Process Reward Models: Calibrated Prediction of Future Rewards via Conditional Optimal Transport
Rachel Ma, Dylan Hadfield-Menell, Kristjan Greenewald
Inference-time scaling methods rely on Process Reward Models (PRMs), which are often poorly calibrated and overestimate success probabilities. We propose, to our knowledge, the fir…
Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMs
Abhay Sheshadri, Aidan Ewart, Phillip Guo +8
Large language models (LLMs) can often be made to behave in undesirable ways that they are explicitly fine-tuned not to. For example, the LLM red-teaming literature has produced a…
The SaTML '24 CNN Interpretability Competition: New Innovations for Concept-Level Interpretability
Stephen Casper, Jieun Yun, Joonhyuk Baek +13
Interpretability techniques are valuable for helping humans understand and oversee AI systems. The SaTML 2024 CNN Interpretability Competition solicited novel methods for studying…
Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF
Anand Siththaranjan, Cassidy Laidlaw, Dylan Hadfield-Menell
In practice, preference learning from human feedback depends on incomplete data with hidden context. Hidden context refers to data that affects the feedback received, but which is…
Red Teaming Deep Neural Networks with Feature Synthesis Tools
Stephen Casper, Yuxiao Li, Jiawei Li +4
Interpretable AI tools are often motivated by the goal of understanding model behavior in out-of-distribution (OOD) contexts. Despite the attention this area of study receives, the…
Diagnostics for Deep Neural Networks with Automated Copy/Paste Attacks
Stephen Casper, Kaivalya Hariharan, Dylan Hadfield-Menell
This paper considers the problem of helping humans exercise scalable oversight over deep neural networks (DNNs). Adversarial examples can be useful by helping to reveal weaknesses…
Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks
Tilman Räuker, Anson Ho, Stephen Casper +1
The last decade of machine learning has seen drastic increases in scale and capabilities. Deep neural networks (DNNs) are increasingly being deployed in the real world. However, th…
Estimating and Penalizing Induced Preference Shifts in Recommender Systems
Micah Carroll, Anca Dragan, Stuart Russell +1
The content that a recommender system (RS) shows to users influences them. Therefore, when choosing a recommender to deploy, one is implicitly also choosing to induce specific inte…
Robust Feature-Level Adversaries are Interpretability Tools
Stephen Casper, Max Nadeau, Dylan Hadfield-Menell +1
The literature on adversarial attacks in computer vision typically focuses on pixel-level perturbations. These tend to be very difficult to interpret. Recent work that manipulates…
An Extensible Interactive Interface for Agent Design
Matthew Rahtz, James Fang, Anca D. Dragan +1
In artificial intelligence, we often specify tasks through a reward function. While this works well in some settings, many tasks are hard to specify this way. In deep reinforcement…
Adversarial Training with Voronoi Constraints
Marc Khoury, Dylan Hadfield-Menell
Adversarial examples are a pervasive phenomenon of machine learning models where seemingly imperceptible perturbations to the input lead to misclassifications for otherwise statist…
The Assistive Multi-Armed Bandit
Lawrence Chan, Dylan Hadfield-Menell, Siddhartha Srinivasa +1
Learning preferences implicit in the choices humans make is a well studied problem in both economics and computer science. However, most work makes the assumption that humans are a…
On the Geometry of Adversarial Examples
Marc Khoury, Dylan Hadfield-Menell
Adversarial examples are a pervasive phenomenon of machine learning models where seemingly imperceptible perturbations to the input lead to misclassifications for otherwise statist…
Active Inverse Reward Design
Sören Mindermann, Rohin Shah, Adam Gleave +1
Designers of AI agents often iterate on the reward function in a trial-and-error process until they get the desired behavior, but this only guarantees good behavior in the training…