VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning

Abstract

Trading off exploration and exploitation in an unknown environment is key to maximising expected return during learning. A Bayes-optimal policy, which does so optimally, conditions its actions not only on the environment state but on the agent's uncertainty about the environment. Computing a Bayes-optimal policy is however intractable for all but the smallest tasks. In this paper, we introduce variational Bayes-Adaptive Deep RL (variBAD), a way to meta-learn to perform approximate inference in an unknown environment, and incorporate task uncertainty directly during action selection. In a grid-world domain, we illustrate how variBAD performs structured online exploration as a function of task uncertainty. We further evaluate variBAD on MuJoCo domains widely used in meta-RL and show that it achieves higher online return than existing methods.

arXiv commentPublished at ICLR 2020

References in corpus (19)

• Auto-Encoding Variational Bayes
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• RL${}^2$: Fast Reinforcement Learning via Slow Reinforcement Learning
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• (More) Efficient Reinforcement Learning via Posterior Sampling
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Fast Context Adaptation via Meta-Learning
• Meta-Reinforcement Learning of Structured Exploration Strategies
• Contextual Decision Processes with Low Bellman Rank are PAC-Learnable
• Learning to Learn: Meta-Critic Networks for Sample Efficient Learning
• Efficient Bayes-Adaptive Reinforcement Learning using Sample-Based Search
• Some Considerations on Learning to Explore via Meta-Reinforcement Learning
• Meta reinforcement learning as task inference
• Learning is planning: near Bayes-optimal reinforcement learning via Monte-Carlo tree search
• Meta-learning of Sequential Strategies
• Policy Certificates: Towards Accountable Reinforcement Learning
• Meta-Amortized Variational Inference and Learning
• Variational Inference for Data-Efficient Model Learning in POMDPs
• Efficient transfer learning and online adaptation with latent variable models for continuous control

Cited by in corpus (33)

• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Infusing model predictive control into meta-reinforcement learning for mobile robots in dynamic environments
• Offline Meta-Reinforcement Learning with Advantage Weighting
• ROI-Constrained Bidding via Curriculum-Guided Bayesian Reinforcement Learning
• Rapidly Adaptable Legged Robots via Evolutionary Meta-Learning
• Multi-task Batch Reinforcement Learning with Metric Learning
• Context Meta-Reinforcement Learning via Neuromodulation
• Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability
• One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL
• MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Exploration
• Offline Meta Learning of Exploration
• Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environment
• Towards Effective Context for Meta-Reinforcement Learning: an Approach based on Contrastive Learning
• Offline Meta-Reinforcement Learning with Online Self-Supervision
• Causal Curiosity: RL Agents Discovering Self-supervised Experiments for Causal Representation Learning
• Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices
• BIMRL: Brain Inspired Meta Reinforcement Learning
• Sequential Topological Representations for Predictive Models of Deformable Objects
• Agent Modelling under Partial Observability for Deep Reinforcement Learning
• Meta-Model-Based Meta-Policy Optimization
• Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture
• Meta Learning MPC using Finite-Dimensional Gaussian Process Approximations
• Exploration in Approximate Hyper-State Space for Meta Reinforcement Learning
• MELD: Meta-Reinforcement Learning from Images via Latent State Models
• Hindsight Foresight Relabeling for Meta-Reinforcement Learning
• Online Limited Memory Neural-Linear Bandits with Likelihood Matching
• Uniform Priors for Data-Efficient Transfer
• Learning MDPs from Features: Predict-Then-Optimize for Sequential Decision Problems by Reinforcement Learning
• Adaptation of Quadruped Robot Locomotion with Meta-Learning
• GalilAI: Out-of-Task Distribution Detection using Causal Active Experimentation for Safe Transfer RL
• HMRL: Hyper-Meta Learning for Sparse Reward Reinforcement Learning Problem
• Estimating Disentangled Belief about Hidden State and Hidden Task for Meta-RL