Reinforced Imitation in Heterogeneous Action Space

Abstract

Imitation learning is an effective alternative approach to learn a policy when the reward function is sparse. In this paper, we consider a challenging setting where an agent and an expert use different actions from each other. We assume that the agent has access to a sparse reward function and state-only expert observations. We propose a method which gradually balances between the imitation learning cost and the reinforcement learning objective. In addition, this method adapts the agent's policy based on either mimicking expert behavior or maximizing sparse reward. We show, through navigation scenarios, that (i) an agent is able to efficiently leverage sparse rewards to outperform standard state-only imitation learning, (ii) it can learn a policy even when its actions are different from the expert, and (iii) the performance of the agent is not bounded by that of the expert, due to the optimized usage of sparse rewards.

The extended version of the work "Reinforced Imitation Learning from Observations" presented on the NeurIPS workshop "Imitation Learning and its Challenges in Robotics"

References in corpus (8)

• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Learning human behaviors from motion capture by adversarial imitation
• Reinforcement and Imitation Learning for Diverse Visuomotor Skills
• Reinforcement Learning from Imperfect Demonstrations
• Generative Adversarial Imitation from Observation
• Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation
• Observational Learning by Reinforcement Learning
• Internal Model from Observations for Reward Shaping

Cited by in corpus (4)

• Making Efficient Use of Demonstrations to Solve Hard Exploration Problems
• Offline Learning from Demonstrations and Unlabeled Experience
• Positive-Unlabeled Reward Learning
• Towards intervention-centric causal reasoning in learning agents