An Imitation from Observation Approach to Transfer Learning with Dynamics Mismatch
arXiv:2008.01594
Abstract
We examine the problem of transferring a policy learned in a source environment to a target environment with different dynamics, particularly in the case where it is critical to reduce the amount of interaction with the target environment during learning. This problem is particularly important in sim-to-real transfer because simulators inevitably model real-world dynamics imperfectly. In this paper, we show that one existing solution to this transfer problem - grounded action transformation - is closely related to the problem of imitation from observation (IfO): learning behaviors that mimic the observations of behavior demonstrations. After establishing this relationship, we hypothesize that recent state-of-the-art approaches from the IfO literature can be effectively repurposed for grounded transfer learning.To validate our hypothesis we derive a new algorithm - generative adversarial reinforced action transformation (GARAT) - based on adversarial imitation from observation techniques. We run experiments in several domains with mismatched dynamics, and find that agents trained with GARAT achieve higher returns in the target environment compared to existing black-box transfer methods
References in corpus (9)
- Trust Region Policy Optimization
- Learning agile and dynamic motor skills for legged robots
- Which Training Methods for GANs do actually Converge?
- Solving Rubik's Cube with a Robot Hand
- Robust Adversarial Reinforcement Learning
- Generative Adversarial Imitation from Observation
- Wasserstein Adversarial Imitation Learning
- Task-Embedded Control Networks for Few-Shot Imitation Learning
- State-only Imitation with Transition Dynamics Mismatch