Fast Adaptation via Policy-Dynamics Value Functions
arXiv:2007.02879
Abstract
Standard RL algorithms assume fixed environment dynamics and require a significant amount of interaction to adapt to new environments. We introduce Policy-Dynamics Value Functions (PD-VF), a novel approach for rapidly adapting to dynamics different from those previously seen in training. PD-VF explicitly estimates the cumulative reward in a space of policies and environments. An ensemble of conventional RL policies is used to gather experience on training environments, from which embeddings of both policies and environments can be learned. Then, a value function conditioned on both embeddings is trained. At test time, a few actions are sufficient to infer the environment embedding, enabling a policy to be selected by maximizing the learned value function (which requires no additional environment interaction). We show that our method can rapidly adapt to new dynamics on a set of MuJoCo domains. Code available at https://github.com/rraileanu/policy-dynamics-value-functions.
References in corpus (29)
- Adam: A Method for Stochastic Optimization
- Dota 2 with Large Scale Deep Reinforcement Learning
- Robots that can adapt like animals
- Human-level performance in first-person multiplayer games with population-based deep reinforcement learning
- Learning to reinforcement learn
- A Study on Overfitting in Deep Reinforcement Learning
- Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
- Modular Multitask Reinforcement Learning with Policy Sketches
- Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
- Distral: Robust Multitask Reinforcement Learning
- Meta-Gradient Reinforcement Learning
- Evolved Policy Gradients
- Meta Reinforcement Learning with Latent Variable Gaussian Processes
- A Dissection of Overfitting and Generalization in Continuous Reinforcement Learning
- Self-Consistent Trajectory Autoencoder: Hierarchical Reinforcement Learning with Trajectory Embeddings
- Learning Parameterized Skills
- Robust Imitation of Diverse Behaviors
- Meta reinforcement learning as task inference
- RIDE: Rewarding Impact-Driven Exploration for Procedurally-Generated Environments
- Decoupling Dynamics and Reward for Transfer Learning
- Universal Successor Features Approximators
- Single Episode Policy Transfer in Reinforcement Learning
- Learning to Compose Skills
- Fast Task Inference with Variational Intrinsic Successor Features
- Learning Shared Representations in Multi-task Reinforcement Learning
- Disentangled Skill Embeddings for Reinforcement Learning
- State2vec: Off-Policy Successor Features Approximators
- VUSFA:Variational Universal Successor Features Approximator to Improve Transfer DRL for Target Driven Visual Navigation
- VPE: Variational Policy Embedding for Transfer Reinforcement Learning