Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables

Abstract

Deep reinforcement learning algorithms require large amounts of experience to learn an individual task. While in principle meta-reinforcement learning (meta-RL) algorithms enable agents to learn new skills from small amounts of experience, several major challenges preclude their practicality. Current methods rely heavily on on-policy experience, limiting their sample efficiency. The also lack mechanisms to reason about task uncertainty when adapting to new tasks, limiting their effectiveness in sparse reward problems. In this paper, we address these challenges by developing an off-policy meta-RL algorithm that disentangles task inference and control. In our approach, we perform online probabilistic filtering of latent task variables to infer how to solve a new task from small amounts of experience. This probabilistic interpretation enables posterior sampling for structured and efficient exploration. We demonstrate how to integrate these task variables with off-policy RL algorithms to achieve both meta-training and adaptation efficiency. Our method outperforms prior algorithms in sample efficiency by 20-100X as well as in asymptotic performance on several meta-RL benchmarks.

Cited by in corpus (99)

• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• Fast Adaptive Task Offloading in Edge Computing based on Meta Reinforcement Learning
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Rapid trial-and-error learning with simulation supports flexible tool use and physical reasoning
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems
• Efficient Exploration via State Marginal Matching
• Learning Invariant Representations for Reinforcement Learning without Reconstruction
• Meta-Learning without Memorization
• Meta-Learning with Warped Gradient Descent
• VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learning
• Meta reinforcement learning as task inference
• Unsupervised Meta-Learning for Reinforcement Learning
• Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement
• Parrot: Data-Driven Behavioral Priors for Reinforcement Learning
• Deep Reinforcement Learning amidst Lifelong Non-Stationarity
• Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learning
• Generalization in Dexterous Manipulation via Geometry-Aware Multi-Task Learning
• Deep Reinforcement Learning in Computer Vision: A Comprehensive Survey
• Single Episode Policy Transfer in Reinforcement Learning
• Infusing model predictive control into meta-reinforcement learning for mobile robots in dynamic environments
• Online Meta-Critic Learning for Off-Policy Actor-Critic Methods
• Meta-Reinforcement Learning Robust to Distributional Shift via Model Identification and Experience Relabeling
• Behavior Self-Organization Supports Task Inference for Continual Robot Learning
• Sample-Efficient Reinforcement Learning via Counterfactual-Based Data Augmentation
• Model-based Adversarial Meta-Reinforcement Learning
• Watch, Try, Learn: Meta-Learning from Demonstrations and Reward
• A Meta-Reinforcement Learning Approach to Process Control
• Offline Meta-Reinforcement Learning with Advantage Weighting
• Rapidly Adaptable Legged Robots via Evolutionary Meta-Learning
• Multi-task Batch Reinforcement Learning with Metric Learning
• Context Meta-Reinforcement Learning via Neuromodulation
• Neural Stochastic Dual Dynamic Programming
• Fast Adaptation via Policy-Dynamics Value Functions
• Learning Adaptive Exploration Strategies in Dynamic Environments Through Informed Policy Regularization
• Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environment
• Offline Meta Learning of Exploration
• MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Exploration
• Curriculum in Gradient-Based Meta-Reinforcement Learning
• Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices
• Evolving Inborn Knowledge For Fast Adaptation in Dynamic POMDP Problems
• Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learning
• Meta Dialogue Policy Learning
• Cross-Trajectory Representation Learning for Zero-Shot Generalization in RL
• Coach-Player Multi-Agent Reinforcement Learning for Dynamic Team Composition
• AdaRL: What, Where, and How to Adapt in Transfer Reinforcement Learning
• Regularizing Meta-Learning via Gradient Dropout
• When Is Generalizable Reinforcement Learning Tractable?
• Learning Mixed-Integer Convex Optimization Strategies for Robot Planning and Control
• Tailoring: encoding inductive biases by optimizing unsupervised objectives at prediction time
• Embedding Adaptation is Still Needed for Few-Shot Learning
• Offline Meta-level Model-based Reinforcement Learning Approach for Cold-Start Recommendation
• A Survey of Exploration Methods in Reinforcement Learning
• Generalized Decision Transformer for Offline Hindsight Information Matching
• Learning Robust State Abstractions for Hidden-Parameter Block MDPs
• Procedural Generalization by Planning with Self-Supervised World Models
• A User's Guide to Calibrating Robotics Simulators
• Meta-Reinforcement Learning in Broad and Non-Parametric Environments
• Agent Modelling under Partial Observability for Deep Reinforcement Learning
• Meta-Model-Based Meta-Policy Optimization
• Linear Representation Meta-Reinforcement Learning for Instant Adaptation
• DOB-Net: Actively Rejecting Unknown Excessive Time-Varying Disturbances
• Generalized Hidden Parameter MDPs Transferable Model-based RL in a Handful of Trials
• Exploration in Approximate Hyper-State Space for Meta Reinforcement Learning
• Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture
• MELD: Meta-Reinforcement Learning from Images via Latent State Models
• Variational Empowerment as Representation Learning for Goal-Based Reinforcement Learning
• Block Contextual MDPs for Continual Learning
• AdaAfford: Learning to Adapt Manipulation Affordance for 3D Articulated Objects via Few-shot Interactions
• OCEAN: Online Task Inference for Compositional Tasks with Context Adaptation
• A Brief Look at Generalization in Visual Meta-Reinforcement Learning
• Off-Policy Meta-Reinforcement Learning Based on Feature Embedding Spaces
• Learning Domain Randomization Distributions for Training Robust Locomotion Policies
• Hindsight Foresight Relabeling for Meta-Reinforcement Learning
• CMML: Contextual Modulation Meta Learning for Cold-Start Recommendation
• Provably Improved Context-Based Offline Meta-RL with Attention and Contrastive Learning
• Improving Context-Based Meta-Reinforcement Learning with Self-Supervised Trajectory Contrastive Learning
• Merging Deterministic Policy Gradient Estimations with Varied Bias-Variance Tradeoff for Effective Deep Reinforcement Learning
• MGHRL: Meta Goal-generation for Hierarchical Reinforcement Learning
• Quick Learner Automated Vehicle Adapting its Roadmanship to Varying Traffic Cultures with Meta Reinforcement Learning
• Context-Based Soft Actor Critic for Environments with Non-stationary Dynamics
• System Identification via Meta-Learning in Linear Time-Varying Environments
• Fighting Copycat Agents in Behavioral Cloning from Observation Histories
• Unifying Gradient Estimators for Meta-Reinforcement Learning via Off-Policy Evaluation
• MAME : Model-Agnostic Meta-Exploration
• Model-based Meta Reinforcement Learning using Graph Structured Surrogate Models
• Recomposing the Reinforcement Learning Building Blocks with Hypernetworks
• SQUIRL: Robust and Efficient Learning from Video Demonstration of Long-Horizon Robotic Manipulation Tasks
• Meta-Reinforcement Learning for Heuristic Planning
• Improving the Generalization of Meta-learning on Unseen Domains via Adversarial Shift
• Connecting Context-specific Adaptation in Humans to Meta-learning
• Estimating Disentangled Belief about Hidden State and Hidden Task for Meta-RL
• GalilAI: Out-of-Task Distribution Detection using Causal Active Experimentation for Safe Transfer RL
• Performance-Weighed Policy Sampling for Meta-Reinforcement Learning
• Active Reinforcement Learning over MDPs
• Neural Auto-Curricula
• Towards Enabling Meta-Learning from Target Models
• Visual Goal-Directed Meta-Learning with Contextual Planning Networks
• When Autonomous Systems Meet Accuracy and Transferability through AI: A Survey
• Adaptation of Quadruped Robot Locomotion with Meta-Learning