Single Episode Policy Transfer in Reinforcement Learning

Abstract

Transfer and adaptation to new unknown environmental dynamics is a key challenge for reinforcement learning (RL). An even greater challenge is performing near-optimally in a single attempt at test time, possibly without access to dense rewards, which is not addressed by current methods that require multiple experience rollouts for adaptation. To achieve single episode transfer in a family of environments with related dynamics, we propose a general algorithm that optimizes a probe and an inference model to rapidly estimate underlying latent variables of test dynamics, which are then immediately used as input to a universal control policy. This modular approach enables integration of state-of-the-art algorithms for variational inference or RL. Moreover, our approach does not require access to rewards at test time, allowing it to perform in settings where existing adaptive approaches cannot. In diverse experimental domains with a single episode test constraint, our method significantly outperforms existing adaptive approaches and shows favorable performance against baselines for robust transfer.

arXiv commentPublished at International Conference on Learning Representations 2020

References in corpus (9)

• Auto-Encoding Variational Bayes
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Fast Context Adaptation via Meta-Learning
• An Overview of Machine Teaching
• Decoupling Dynamics and Reward for Transfer Learning
• Learning to Explore with Meta-Policy Gradient
• Efficient transfer learning and online adaptation with latent variable models for continuous control
• Fingerprint Policy Optimisation for Robust Reinforcement Learning

Cited by in corpus (8)

• Self-Supervised Policy Adaptation during Deployment
• Behavior Self-Organization Supports Task Inference for Continual Robot Learning
• Generalization in Reinforcement Learning by Soft Data Augmentation
• SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies
• Procedural Content Generation: Better Benchmarks for Transfer Reinforcement Learning
• Fast Adaptation via Policy-Dynamics Value Functions
• Unsupervised Visual Attention and Invariance for Reinforcement Learning
• PAC Bounds for Imitation and Model-based Batch Learning of Contextual Markov Decision Processes