Apprenticeship Learning using Inverse Reinforcement Learning and Gradient Methods

Abstract

In this paper we propose a novel gradient algorithm to learn a policy from an expert's observed behavior assuming that the expert behaves optimally with respect to some unknown reward function of a Markovian Decision Problem. The algorithm's aim is to find a reward function such that the resulting optimal policy matches well the expert's observed behavior. The main difficulty is that the mapping from the parameters to policies is both nonsmooth and highly redundant. Resorting to subdifferentials solves the first difficulty, while the second one is over- come by computing natural gradients. We tested the proposed method in two artificial domains and found it to be more reliable and efficient than some previous methods.

Appears in Proceedings of the Twenty-Third Conference on Uncertainty in Artificial Intelligence (UAI2007)

Cited by in corpus (27)

• An Introduction to Deep Reinforcement Learning
• Maximum Entropy Deep Inverse Reinforcement Learning
• Reinforcement Learning for Automatic Test Case Prioritization and Selection in Continuous Integration
• Learning with a Strong Adversary
• Few-shot Autoregressive Density Estimation: Towards Learning to Learn Distributions
• Inverse Reinforcement Learning in Swarm Systems
• Online inverse reinforcement learning with unknown disturbances
• Where Do You Think You're Going?: Inferring Beliefs about Dynamics from Behavior
• Inverse Reinforcement Learning from Summary Data
• Strictly Batch Imitation Learning by Energy-based Distribution Matching
• Inverse Reinforcement Learning via Nonparametric Spatio-Temporal Subgoal Modeling
• Lifelong Inverse Reinforcement Learning
• Deep Inverse Q-learning with Constraints
• Dueling RL: Reinforcement Learning with Trajectory Preferences
• On the Performance of Maximum Likelihood Inverse Reinforcement Learning
• Inverse reinforcement learning for autonomous navigation via differentiable semantic mapping and planning
• Multi-class Generalized Binary Search for Active Inverse Reinforcement Learning
• Bellman Gradient Iteration for Inverse Reinforcement Learning
• A Lower Bound for the Sample Complexity of Inverse Reinforcement Learning
• Behavior Pattern Recognition using A New Representation Model
• Provably Efficient Generative Adversarial Imitation Learning for Online and Offline Setting with Linear Function Approximation
• Interactive Learning of Environment Dynamics for Sequential Tasks
• Learning Navigation Costs from Demonstration in Partially Observable Environments
• Learning Policies for Markov Decision Processes from Data
• Sequence-based Multimodal Apprenticeship Learning For Robot Perception and Decision Making
• Inverse Reinforcement Learning with Gaussian Process
• Online inverse reinforcement learning for nonlinear systems