Meta reinforcement learning as task inference

Abstract

Humans achieve efficient learning by relying on prior knowledge about the structure of naturally occurring tasks. There is considerable interest in designing reinforcement learning (RL) algorithms with similar properties. This includes proposals to learn the learning algorithm itself, an idea also known as meta learning. One formal interpretation of this idea is as a partially observable multi-task RL problem in which task information is hidden from the agent. Such unknown task problems can be reduced to Markov decision processes (MDPs) by augmenting an agent's observations with an estimate of the belief about the task based on past experience. However estimating the belief state is intractable in most partially-observed MDPs. We propose a method that separately learns the policy and the task belief by taking advantage of various kinds of privileged information. Our approach can be very effective at solving standard meta-RL environments, as well as a complex continuous control environment with sparse rewards and requiring long-term memory.

References in corpus (23)

• Representation Learning with Contrastive Predictive Coding
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• A Simple Neural Attentive Meta-Learner
• Emergence of Locomotion Behaviours in Rich Environments
• DeepMind Control Suite
• Learning to reinforcement learn
• Deep Variational Information Bottleneck
• Learning Continuous Control Policies by Stochastic Value Gradients
• Further Optimal Regret Bounds for Thompson Sampling
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• A Study on Overfitting in Deep Reinforcement Learning
• Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
• Meta-Reinforcement Learning of Structured Exploration Strategies
• Grounded Language Learning in a Simulated 3D World
• Unsupervised Predictive Memory in a Goal-Directed Agent
• Reverse Curriculum Generation for Reinforcement Learning
• Emergent Coordination Through Competition
• Gotta Learn Fast: A New Benchmark for Generalization in RL
• Relevant sparse codes with variational information bottleneck
• Neural Predictive Belief Representations
• Guided Meta-Policy Search
• Learning Dynamics Model in Reinforcement Learning by Incorporating the Long Term Future

Cited by in corpus (33)

• Solving Rubik's Cube with a Robot Hand
• Stabilizing Transformers for Reinforcement Learning
• VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
• Meta-Reinforcement Learning Robust to Distributional Shift via Model Identification and Experience Relabeling
• BADGER: Learning to (Learn [Learning Algorithms] through Multi-Agent Communication)
• Inverse Rational Control with Partially Observable Continuous Nonlinear Dynamics
• Model-based Adversarial Meta-Reinforcement Learning
• Behavior Priors for Efficient Reinforcement Learning
• Offline Meta-Reinforcement Learning with Advantage Weighting
• ROI-Constrained Bidding via Curriculum-Guided Bayesian Reinforcement Learning
• Multi-task Batch Reinforcement Learning with Metric Learning
• Bayesian decision-making under misspecified priors with applications to meta-learning
• Context Meta-Reinforcement Learning via Neuromodulation
• Offline Meta Learning of Exploration
• MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Exploration
• Fast Adaptation via Policy-Dynamics Value Functions
• Learning Adaptive Exploration Strategies in Dynamic Environments Through Informed Policy Regularization
• Offline Meta-Reinforcement Learning with Online Self-Supervision
• Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices
• Neural Recursive Belief States in Multi-Agent Reinforcement Learning
• Efficient Transformers in Reinforcement Learning using Actor-Learner Distillation
• Meta Learning Black-Box Population-Based Optimizers
• Meta-Model-Based Meta-Policy Optimization
• Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture
• Block Contextual MDPs for Continual Learning
• Exploration in Approximate Hyper-State Space for Meta Reinforcement Learning
• MELD: Meta-Reinforcement Learning from Images via Latent State Models
• Adaptive Transformers in RL
• Off-Policy Meta-Reinforcement Learning Based on Feature Embedding Spaces
• The Information Geometry of Unsupervised Reinforcement Learning
• OCEAN: Online Task Inference for Compositional Tasks with Context Adaptation
• What is Going on Inside Recurrent Meta Reinforcement Learning Agents?
• Estimating Disentangled Belief about Hidden State and Hidden Task for Meta-RL