Off-Dynamics Reinforcement Learning: Training for Transfer with Domain Classifiers

Abstract

We propose a simple, practical, and intuitive approach for domain adaptation in reinforcement learning. Our approach stems from the idea that the agent's experience in the source domain should look similar to its experience in the target domain. Building off of a probabilistic view of RL, we formally show that we can achieve this goal by compensating for the difference in dynamics by modifying the reward function. This modified reward function is simple to estimate by learning auxiliary classifiers that distinguish source-domain transitions from target-domain transitions. Intuitively, the modified reward function penalizes the agent for visiting states and taking actions in the source domain which are not possible in the target domain. Said another way, the agent is penalized for transitions that would indicate that the agent is interacting with the source domain, rather than the target domain. Our approach is applicable to domains with continuous states and actions and does not require learning an explicit model of the dynamics. On discrete and continuous control tasks, we illustrate the mechanics of our approach and demonstrate its scalability to high-dimensional tasks.

arXiv commentPublished at ICLR 2021. Code (https://github.com/google-research/google-research/tree/master/darc) and blog post (https://blog.ml.cmu.edu/2020/07/31/maintaining-the-illusion-of-reality-transfer-in-rl-by-keeping-agents-in-the-darc)

References in corpus (30)

• Adam: A Method for Stochastic Optimization
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• A Simple Neural Attentive Meta-Learner
• FCNs in the Wild: Pixel-level Adversarial and Constraint-based Adaptation
• A review of domain adaptation without target labels
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review
• Safe Model-based Reinforcement Learning with Stability Guarantees
• Path integrals and symmetry breaking for optimal control theory
• Doubly Robust Policy Evaluation and Learning
• Domain Adaptation for Visual Applications: A Comprehensive Survey
• Off-Policy Deep Reinforcement Learning without Exploration
• Maximum a Posteriori Policy Optimisation
• Learning in Implicit Generative Models
• Amortised MAP Inference for Image Super-resolution
• EPOpt: Learning Robust Neural Network Policies Using Model Ensembles
• DARLA: Improving Zero-Shot Transfer in Reinforcement Learning
• Detecting and Correcting for Label Shift with Black Box Predictors
• Variational Inference using Implicit Distributions
• Model Predictive Path Integral Control using Covariance Variable Importance Sampling
• Agnostic System Identification for Model-Based Reinforcement Learning
• Transfer Learning for Related Reinforcement Learning Tasks via Image-to-Image Translation
• Generative Adversarial Nets from a Density Ratio Estimation Perspective
• Preparing for the Unknown: Learning a Universal Policy with Online System Identification
• V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control
• Scaling Up Robust MDPs by Reinforcement Learning
• Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning
• Mutual Alignment Transfer Learning
• Universal Value Density Estimation for Imitation Learning and Goal-Conditioned Reinforcement Learning
• Cycle-Consistent Adversarial Learning as Approximate Bayesian Inference

Cited by in corpus (6)

• SimGAN: Hybrid Simulator Identification for Domain Adaptation via Adversarial Reinforcement Learning
• Inverse Constrained Reinforcement Learning
• Unsupervised Domain Adaptation with Dynamics-Aware Rewards in Reinforcement Learning
• Fractional Transfer Learning for Deep Model-Based Reinforcement Learning
• Discriminator Augmented Model-Based Reinforcement Learning
• Off-Dynamics Inverse Reinforcement Learning from Hetero-Domain