Cooperative Inverse Reinforcement Learning
arXiv:1606.03137
Abstract
For an autonomous system to be helpful to humans and to pose no unwarranted risks, it needs to align its values with those of the humans in its environment in such a way that its actions contribute to the maximization of value for the humans. We propose a formal definition of the value alignment problem as cooperative inverse reinforcement learning (CIRL). A CIRL problem is a cooperative, partial-information game with two agents, human and robot; both are rewarded according to the human's reward function, but the robot does not initially know what this is. In contrast to classical IRL, where the human is assumed to act optimally in isolation, optimal CIRL solutions produce behaviors such as active teaching, active learning, and communicative actions that are more effective in achieving value alignment. We show that computing optimal joint policies in CIRL games can be reduced to solving a POMDP, prove that optimality in isolation is suboptimal in CIRL, and derive an approximate CIRL algorithm.
References in corpus (1)
Cited by in corpus (78)
- Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications
- Artificial Intelligence, Values and Alignment
- Deep Reinforcement Learning: An Overview
- An Algorithmic Perspective on Imitation Learning
- Formulation of Deep Reinforcement Learning Architecture Toward Autonomous Driving for On-Ramp Merge
- AI Safety Gridworlds
- A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms
- Unsolved Problems in ML Safety
- Game-Theoretic Modeling of Human Adaptation in Human-Robot Collaboration
- Machine Theory of Mind
- Modeling Others using Oneself in Multi-Agent Reinforcement Learning
- Inverse Reward Design
- Reinforcement Learning for Generative AI: State of the Art, Opportunities and Open Research Challenges
- Safe Imitation Learning via Fast Bayesian Reward Inference from Preferences
- Policy Teaching via Environment Poisoning: Training-time Adversarial Attacks against Reinforcement Learning
- Bridging adaptive management and reinforcement learning for more robust decisions
- Beyond Preferences in AI Alignment
- Robot Planning with Mathematical Models of Human State and Action
- Inverse Filtering for Hidden Markov Models with Applications to Counter-Adversarial Autonomous Systems
- Multi-agent Inverse Reinforcement Learning for Certain General-sum Stochastic Games
- A Survey on Deep Learning Toolkits and Libraries for Intelligent User Interfaces
- Learner-aware Teaching: Inverse Reinforcement Learning with Preferences and Constraints
- Quantifying Hypothesis Space Misspecification in Learning from Human-Robot Demonstrations and Physical Corrections
- Penalizing side effects using stepwise relative reachability
- Algorithms for the Greater Good! On Mental Modeling and Acceptable Symbiosis in Human-AI Collaboration
- Enabling Value Sensitive AI Systems through Participatory Design Fictions
- Feature Expansive Reward Learning: Rethinking Human Input
- A mathematical theory of cooperative communication
- Using Human Ratings for Feedback Control: A Supervised Learning Approach with Application to Rehabilitation Robotics
- Avoiding Side Effects By Considering Future Tasks
- Impossibility and Uncertainty Theorems in AI Value Alignment (or why your AGI should not have a utility function)
- Balancing Explicability and Explanation in Human-Aware Planning
- Conservative Agency via Attainable Utility Preservation
- Axes for Sociotechnical Inquiry in AI Research
- Recent Advances in Leveraging Human Guidance for Sequential Decision-Making Tasks
- AvE: Assistance via Empowerment
- adaPARL: Adaptive Privacy-Aware Reinforcement Learning for Sequential-Decision Making Human-in-the-Loop Systems
- Incentives for Responsiveness, Instrumental Control and Impact
- Good and safe uses of AI Oracles
- Human-in-the-Loop Imitation Learning using Remote Teleoperation
- Learning Norms from Stories: A Prior for Value Aligned Agents
- Towards Deployment of Robust AI Agents for Human-Machine Partnerships
- Morality, Machines and the Interpretation Problem: A Value-based, Wittgensteinian Approach to Building Moral Agents
- Medical Dead-ends and Learning to Identify High-risk States and Treatments
- Optimization's Neglected Normative Commitments
- Inverse Dynamic Games Based on Maximum Entropy Inverse Reinforcement Learning
- Learning from My Partner's Actions: Roles in Decentralized Robot Teams
- Parallel Knowledge Transfer in Multi-Agent Reinforcement Learning
- Learning to Communicate Implicitly By Actions
- Human irrationality: both bad and good for reward inference
- Toward negotiable reinforcement learning: shifting priorities in Pareto optimal sequential decision-making
- Sequential Cooperative Bayesian Inference
- Teaching Social Behavior through Human Reinforcement for Ad hoc Teamwork -The STAR Framework
- Reward Poisoning in Reinforcement Learning: Attacks Against Unknown Learners in Unknown Environments
- Forming Real-World Human-Robot Cooperation for Tasks With General Goal
- What Would Jiminy Cricket Do? Towards Agents That Behave Morally
- Parenting: Safe Reinforcement Learning from Human Input
- Interaction-limited Inverse Reinforcement Learning
- Multi-Principal Assistance Games
- Off-Policy Exploitability-Evaluation in Two-Player Zero-Sum Markov Games
- Morphology dictates a robot's ability to ground crowd-proposed language
- Explaining Reward Functions to Humans for Better Human-Robot Collaboration
- The Challenge of Value Alignment: from Fairer Algorithms to AI Safety
- Classification Under Human Assistance
- Continuous-Time Inverse Quadratic Optimal Control Problem
- Interactive Inverse Reinforcement Learning for Cooperative Games
- AGENT: A Benchmark for Core Psychological Reasoning
- Human-Interactive Subgoal Supervision for Efficient Inverse Reinforcement Learning
- Mammalian Value Systems
- Chess as a Testing Grounds for the Oracle Approach to AI Safety
- The Goofus & Gallant Story Corpus for Practical Value Alignment
- Assisted Robust Reward Design
- Value Alignment Verification
- Regression Under Human Assistance
- Mutual Reinforcement Learning
- Generalizing the theory of cooperative inference
- Robustness to fundamental uncertainty in AGI alignment
- Towards an unanimous international regulatory body for responsible use of Artificial Intelligence [UIRB-AI]