Inverse Reward Design
arXiv:1711.02827
Abstract
Autonomous agents optimize the reward function we give them. What they don't know is how hard it is for us to design a reward function that actually captures what we want. When designing the reward, we might think of some specific training scenarios, and make sure that the reward will lead to the right behavior in those scenarios. Inevitably, agents encounter new scenarios (e.g., new types of terrain) where optimizing that same reward may lead to undesired behavior. Our insight is that reward functions are merely observations about what the designer actually wants, and that they should be interpreted in the context in which they were designed. We introduce inverse reward design (IRD) as the problem of inferring the true objective based on the designed reward and the training MDP. We introduce approximate methods for solving IRD problems, and use their solution to plan risk-averse behavior in test MDPs. Empirical results suggest that this approach can help alleviate negative side effects of misspecified reward functions and mitigate reward hacking.
Advances in Neural Information Processing Systems 30 (NIPS 2017) Revised Oct 2020 to fix a typo in Eq. 3
References in corpus (2)
Cited by in corpus (22)
- Deep Reinforcement Learning: An Overview
- Challenges of Real-World Reinforcement Learning
- Scalable agent alignment via reward modeling: a research direction
- AI Safety Gridworlds
- Modeling Others using Oneself in Multi-Agent Reinforcement Learning
- Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definition
- An empirical investigation of the challenges of real-world reinforcement learning
- Deep Imitation Learning for Autonomous Driving in Generic Urban Scenarios with Enhanced Safety
- Maximum Entropy RL (Provably) Solves Some Robust RL Problems
- Safe Option-Critic: Learning Safety in the Option-Critic Architecture
- Reinforcement Learning with Perturbed Rewards
- Including Uncertainty when Learning from Human Corrections
- Active Inverse Reward Design
- Choice Set Misspecification in Reward Inference
- Parenting: Safe Reinforcement Learning from Human Input
- Towards Mixed Optimization for Reinforcement Learning with Program Synthesis
- Avoiding Side Effects in Complex Environments
- Legible Normativity for AI Alignment: The Value of Silly Rules
- Balancing Performance and Human Autonomy with Implicit Guidance Agent
- Minimizing the Negative Side Effects of Planning with Reduced Models
- Relational Mimic for Visual Adversarial Imitation Learning
- Estimating Rationally Inattentive Utility Functions with Deep Clustering for Framing - Applications in YouTube Engagement Dynamics