Transfer Learning for Related Reinforcement Learning Tasks via Image-to-Image Translation

Abstract

Despite the remarkable success of Deep RL in learning control policies from raw pixels, the resulting models do not generalize. We demonstrate that a trained agent fails completely when facing small visual changes, and that fine-tuning---the common transfer learning paradigm---fails to adapt to these changes, to the extent that it is faster to re-train the model from scratch. We show that by separating the visual transfer task from the control policy we achieve substantially better sample efficiency and transfer behavior, allowing an agent trained on the source task to transfer well to the target tasks. The visual mapping from the target to the source domain is performed using unaligned GANs, resulting in a control policy that can be further improved using imitation learning from imperfect demonstrations. We demonstrate the approach on synthetic visual variants of the Breakout game, as well as on transfer between subsequent levels of Road Fighter, a Nintendo car-driving game. A visualization of our approach can be seen in https://youtu.be/4mnkzYyXMn4 and https://youtu.be/KCGTrQi6Ogo .

arXiv commentProceedings of the 36th International Conference on Machine Learning (ICML 2019)

References in corpus (16)

• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Playing Atari with Deep Reinforcement Learning
• How transferable are features in deep neural networks?
• Prioritized Experience Replay
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• CyCADA: Cycle-Consistent Adversarial Domain Adaptation
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Unsupervised Image-to-Image Translation Networks
• Generative Adversarial Imitation Learning
• A Study on Overfitting in Deep Reinforcement Learning
• Schema Networks: Zero-shot Transfer with a Generative Causal Model of Intuitive Physics
• Self-Imitation Learning
• DARLA: Improving Zero-Shot Transfer in Reinforcement Learning
• Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks
• One-Shot Reinforcement Learning for Robot Navigation with Interactive Replay
• Semantically Decomposing the Latent Spaces of Generative Adversarial Networks

Cited by in corpus (19)

• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Generative Adversarial Networks (GANs) in Networking: A Comprehensive Survey & Evaluation
• Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
• Image-to-Image Translation: Methods and Applications
• MixStyle Neural Networks for Domain Generalization and Adaptation
• Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers
• Procedural Content Generation: Better Benchmarks for Transfer Reinforcement Learning
• Reinforcement Learning Generalization with Surprise Minimization
• Scheduling Out-of-Coverage Vehicular Communications Using Reinforcement Learning
• Robust Visual Domain Randomization for Reinforcement Learning
• Mutual Information Based Knowledge Transfer Under State-Action Dimension Mismatch
• Domain Adaptation In Reinforcement Learning Via Latent Unified State Representation
• Decoupling Value and Policy for Generalization in Reinforcement Learning
• Transferable Active Grasping and Real Embodied Dataset
• Visual Analogies between Atari Games for Studying Transfer Learning in RL
• Continuous Control with Contexts, Provably
• Component Transfer Learning for Deep RL Based on Abstract Representations
• Few-Shot Image-to-Semantics Translation for Policy Transfer in Reinforcement Learning
• BERT2DNN: BERT Distillation with Massive Unlabeled Data for Online E-Commerce Search