DARLA: Improving Zero-Shot Transfer in Reinforcement Learning

Abstract

Domain adaptation is an important open problem in deep reinforcement learning (RL). In many scenarios of interest data is hard to obtain, so agents may learn a source policy in a setting where data is readily available, with the hope that it generalises well to the target domain. We propose a new multi-stage RL agent, DARLA (DisentAngled Representation Learning Agent), which learns to see before learning to act. DARLA's vision is based on learning a disentangled representation of the observed environment. Once DARLA can see, it is able to acquire source policies that are robust to many domain shifts - even with no access to the target domain. DARLA significantly outperforms conventional baselines in zero-shot domain adaptation scenarios, an effect that holds across a variety of RL environments (Jaco arm, DeepMind Lab) and base RL algorithms (DQN, A3C and EC).

ICML 2017

References in corpus (12)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Asynchronous Methods for Deep Reinforcement Learning
• Deep Convolutional Inverse Graphics Network
• Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• Discovering Hidden Factors of Variation in Deep Networks
• Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning
• Disentangling Factors of Variation via Generative Entangling
• High-Dimensional Probability Estimation with Deep Density Models
• A Survey of Inductive Biases for Factorial Representation-Learning
• Understanding Visual Concepts with Continuation Learning
• Learning Transferable Policies for Monocular Reactive MAV Control

Cited by in corpus (58)

• An Introduction to Deep Reinforcement Learning
• Visual Reinforcement Learning with Imagined Goals
• State Representation Learning for Control: An Overview
• Multi-Object Representation Learning with Iterative Variational Inference
• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Unsupervised deep learning identifies semantic disentanglement in single inferotemporal neurons
• Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task
• Weakly-Supervised Disentanglement Without Compromises
• Universal Planning Networks
• Gotta Learn Fast: A New Benchmark for Generalization in RL
• Learning Disentangled Joint Continuous and Discrete Representations
• Transfer Learning for Related Reinforcement Learning Tasks via Image-to-Image Translation
• Disentangling Factors of Variation Using Few Labels
• Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learning
• IB-GAN: Disentangled Representation Learning with Information Bottleneck Generative Adversarial Networks
• Mid-Level Visual Representations Improve Generalization and Sample Efficiency for Learning Visuomotor Policies
• Unsupervised Model Selection for Variational Disentangled Representation Learning
• Visual Semantic Navigation using Scene Priors
• Unsupervised State Representation Learning in Atari
• Internet Congestion Control via Deep Reinforcement Learning
• Structure in Deep Reinforcement Learning: A Survey and Open Problems
• Towards Privacy-Preserving Visual Recognition via Adversarial Training: A Pilot Study
• Continual State Representation Learning for Reinforcement Learning using Generative Replay
• Dynamics-aware Embeddings
• Shaping Belief States with Generative Environment Models for RL
• Dual Swap Disentangling
• D2RL: Deep Dense Architectures in Reinforcement Learning
• Hierarchically Organized Latent Modules for Exploratory Search in Morphogenetic Systems
• Self-Attentional Credit Assignment for Transfer in Reinforcement Learning
• Learning to Navigate Using Mid-Level Visual Priors
• Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers
• Learning Group Structure and Disentangled Representations of Dynamical Environments
• Hierarchical Policy Learning is Sensitive to Goal Space Design
• From Few to More: Large-scale Dynamic Multiagent Curriculum Learning
• On the Fairness of Disentangled Representations
• Disentangling and Learning Robust Representations with Natural Clustering
• ToyArchitecture: Unsupervised Learning of Interpretable Models of the World
• Disentangled Cumulants Help Successor Representations Transfer to New Tasks
• Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings
• MRAC-RL: A Framework for On-Line Policy Adaptation Under Parametric Model Uncertainty
• Product Kanerva Machines: Factorized Bayesian Memory
• On the Power of Multitask Representation Learning in Linear MDP
• Domain Adaptation In Reinforcement Learning Via Latent Unified State Representation
• Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning
• Unsupervised Domain Adaptation with Dynamics-Aware Rewards in Reinforcement Learning
• Self-supervised Visual Reinforcement Learning with Object-centric Representations
• Zero-Shot Learning of Text Adventure Games with Sentence-Level Semantics
• Coordinated Heterogeneous Distributed Perception based on Latent Space Representation
• Generalization to Novel Objects using Prior Relational Knowledge
• On the Latent Holes of VAEs for Text Generation
• Making Curiosity Explicit in Vision-based RL
• Towards a Theoretical Understanding of the Robustness of Variational Autoencoders
• Certifiably Robust Variational Autoencoders
• Pretrained Encoders are All You Need
• Provably Efficient Third-Person Imitation from Offline Observation
• Extracting Latent State Representations with Linear Dynamics from Rich Observations
• Towards Better Understanding of Disentangled Representations via Mutual Information
• Discovering Influential Factors in Variational Autoencoders