Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch

Abstract

We study the inverse reinforcement learning (IRL) problem under a transition dynamics mismatch between the expert and the learner. Specifically, we consider the Maximum Causal Entropy (MCE) IRL learner model and provide a tight upper bound on the learner's performance degradation based on the ${\ell }_1$-distance between the transition dynamics of the expert and the learner. Leveraging insights from the Robust RL literature, we propose a robust MCE IRL algorithm, which is a principled approach to help with this mismatch. Finally, we empirically demonstrate the stable performance of our algorithm compared to the standard MCE IRL algorithm under transition dynamics mismatches in both finite and continuous MDP problems.

References in corpus (16)

• Continuous control with deep reinforcement learning
• Trust Region Policy Optimization
• Robust Adversarial Reinforcement Learning
• Maximum Entropy Deep Inverse Reinforcement Learning
• Challenges of Real-World Reinforcement Learning
• Reinforcement Learning in Healthcare: A Survey
• A Theory of Regularized Markov Decision Processes
• Generative Adversarial Imitation from Observation
• Action Robust Reinforcement Learning and Applications in Continuous Control
• Robust Reinforcement Learning for Continuous Control with Model Misspecification
• Teaching Inverse Reinforcement Learners via Features and Demonstrations
• Provably Efficient Imitation Learning from Observation Alone
• Dual Policy Iteration
• Deep Robust Kalman Filter
• State Alignment-based Imitation Learning
• State-only Imitation with Transition Dynamics Mismatch

Cited by in corpus (1)

• Unsupervised Domain Adaptation with Dynamics-Aware Rewards in Reinforcement Learning