Off-Dynamics Inverse Reinforcement Learning from Hetero-Domain

Abstract

We propose an approach for inverse reinforcement learning from hetero-domain which learns a reward function in the simulator, drawing on the demonstrations from the real world. The intuition behind the method is that the reward function should not only be oriented to imitate the experts, but should encourage actions adjusted for the dynamics difference between the simulator and the real world. To achieve this, the widely used GAN-inspired IRL method is adopted, and its discriminator, recognizing policy-generating trajectories, is modified with the quantification of dynamics difference. The training process of the discriminator can yield the transferable reward function suitable for simulator dynamics, which can be guaranteed by derivation. Effectively, our method assigns higher rewards for demonstration trajectories which do not exploit discrepancies between the two domains. With extensive experiments on continuous control tasks, our method shows its effectiveness and demonstrates its scalability to high-dimensional tasks.

References in corpus (8)

• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• Control of Memory, Active Perception, and Action in Minecraft
• One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning
• Meta-Inverse Reinforcement Learning with Probabilistic Context Variables
• Domain Adaptive Imitation Learning
• Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers
• State-only Imitation with Transition Dynamics Mismatch
• Cross-domain Imitation from Observations