Inverse Constrained Reinforcement Learning

Abstract

In real world settings, numerous constraints are present which are hard to specify mathematically. However, for the real world deployment of reinforcement learning (RL), it is critical that RL agents are aware of these constraints, so that they can act safely. In this work, we consider the problem of learning constraints from demonstrations of a constraint-abiding agent's behavior. We experimentally validate our approach and show that our framework can successfully learn the most likely constraints that the agent respects. We further show that these learned constraints are \textit{transferable} to new agents that may have different morphologies and/or reward functions. Previous works in this regard have either mainly been restricted to tabular (discrete) settings, specific types of constraints or assume the environment's transition dynamics. In contrast, our framework is able to learn arbitrary \textit{Markovian} constraints in high-dimensions in a completely model-free setting. The code can be found it: \url{https://github.com/shehryar-malik/icrl}.

arXiv commentCamera-ready version for ICML 2021

References in corpus (10)

• Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
• Reward Constrained Policy Optimization
• Scalable agent alignment via reward modeling: a research direction
• AI Safety Gridworlds
• Interactive Learning from Policy-Dependent Human Feedback
• Reinforcement Learning with Convex Constraints
• Learning Constraints from Demonstrations
• Preferences Implicit in the State of the World
• Inferring geometric constraints in human demonstrations
• Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers