Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environment
arXiv:2104.05632
Abstract
Reinforcement learning from large-scale offline datasets provides us with the ability to learn policies without potentially unsafe or impractical exploration. Significant progress has been made in the past few years in dealing with the challenge of correcting for differing behavior between the data collection and learned policies. However, little attention has been paid to potentially changing dynamics when transferring a policy to the online setting, where performance can be up to 90% reduced for existing methods. In this paper we address this problem with Augmented World Models (AugWM). We augment a learned dynamics model with simple transformations that seek to capture potential changes in physical properties of the robot, leading to more robust policies. We not only train our policy in this new setting, but also provide it with the sampled augmentation as a context, allowing it to adapt to changes in the environment. At test time we learn the context in a self-supervised fashion by approximating the augmentation which corresponds to the new environment. We rigorously evaluate our approach on over 100 different changed dynamics settings, and show that this simple approach can significantly improve the zero-shot generalization of a recent state-of-the-art baseline, often achieving successful policies where the baseline fails.
arXiv commentAccepted @ ICML 2021; Spotlight @ ICLR 2021 "Self-Supervision for Reinforcement Learning Workshop"
References in corpus (26)
- Soft Actor-Critic Algorithms and Applications
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
- Solving Rubik's Cube with a Robot Hand
- Conservative Q-Learning for Offline Reinforcement Learning
- Recurrent World Models Facilitate Policy Evolution
- Learning Latent Dynamics for Planning from Pixels
- Continuous Deep Q-Learning with Model-based Acceleration
- Model-Ensemble Trust-Region Policy Optimization
- Challenges of Real-World Reinforcement Learning
- Behavior Regularized Offline Reinforcement Learning
- MOPO: Model-based Offline Policy Optimization
- Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
- Model-Based Reinforcement Learning via Meta-Policy Optimization
- VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
- Automatic Data Augmentation for Generalization in Deep Reinforcement Learning
- Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL
- Hidden Parameter Markov Decision Processes: A Semiparametric Regression Approach for Discovering Latent Task Parametrizations
- Meta-Learning Requires Meta-Augmentation
- Environment Probing Interaction Policies
- Reinforcement Learning for Pivoting Task
- Benchmark Environments for Multitask Learning in Continuous Domains
- Transient Non-Stationarity and Generalisation in Deep Reinforcement Learning
- Ready Policy One: World Building Through Active Learning
- Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learning
- Linear Representation Meta-Reinforcement Learning for Instant Adaptation
- On Pathologies in KL-Regularized Reinforcement Learning from Expert Demonstrations