Meta Reinforcement Learning with Latent Variable Gaussian Processes

Abstract

Learning from small data sets is critical in many practical applications where data collection is time consuming or expensive, e.g., robotics, animal experiments or drug design. Meta learning is one way to increase the data efficiency of learning algorithms by generalizing learned concepts from a set of training tasks to unseen, but related, tasks. Often, this relationship between tasks is hard coded or relies in some other way on human expertise. In this paper, we frame meta learning as a hierarchical latent variable model and infer the relationship between tasks automatically from data. We apply our framework in a model-based reinforcement learning setting and show that our meta-learning model effectively generalizes to novel tasks by identifying how new tasks relate to prior ones from minimal data. This results in up to a 60% reduction in the average interaction time needed to solve tasks compared to strong baselines.

arXiv comment11 pages, 7 figures

References in corpus (7)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Robots that can adapt like animals
• Gaussian Processes for Data-Efficient Learning in Robotics and Control
• Gaussian Processes for Big Data
• Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Control
• Hidden Parameter Markov Decision Processes: A Semiparametric Regression Approach for Discovering Latent Task Parametrizations
• Latent Gaussian Process Regression

Cited by in corpus (36)

• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Assessing Generalization in Deep Reinforcement Learning
• Model-Based Meta-Reinforcement Learning for Flight with Suspended Payloads
• Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning
• Causal Induction from Visual Observations for Goal Directed Tasks
• Deep Reinforcement Learning in Computer Vision: A Comprehensive Survey
• Infusing model predictive control into meta-reinforcement learning for mobile robots in dynamic environments
• Meta-Reinforcement Learning Robust to Distributional Shift via Model Identification and Experience Relabeling
• Watch, Try, Learn: Meta-Learning from Demonstrations and Reward
• Model-based Adversarial Meta-Reinforcement Learning
• Offline Meta-Reinforcement Learning with Advantage Weighting
• Meta-Learning surrogate models for sequential decision making
• Complementary Meta-Reinforcement Learning for Fault-Adaptive Control
• Multi-task Batch Reinforcement Learning with Metric Learning
• Fast Adaptation via Policy-Dynamics Value Functions
• Towards Deployment of Robust AI Agents for Human-Machine Partnerships
• Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrifices
• Context-Aware Safe Reinforcement Learning for Non-Stationary Environments
• Probabilistic Active Meta-Learning
• Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learning
• How Does an Approximate Model Help in Reinforcement Learning?
• Meta-Model-Based Meta-Policy Optimization
• DOB-Net: Actively Rejecting Unknown Excessive Time-Varying Disturbances
• Learning State Abstractions for Transfer in Continuous Control
• Meta Learning MPC using Finite-Dimensional Gaussian Process Approximations
• Generalized Hidden Parameter MDPs Transferable Model-based RL in a Handful of Trials
• Adaptive Adversarial Training for Meta Reinforcement Learning
• PerSim: Data-Efficient Offline Reinforcement Learning with Heterogeneous Agents via Personalized Simulators
• Improving Context-Based Meta-Reinforcement Learning with Self-Supervised Trajectory Contrastive Learning
• Model-based Meta Reinforcement Learning using Graph Structured Surrogate Models
• Meta-Reinforcement Learning for Adaptive Motor Control in Changing Robot Dynamics and Environments
• HMRL: Hyper-Meta Learning for Sparse Reward Reinforcement Learning Problem
• Improving the Robustness of Reinforcement Learning Policies with ${\mathcal{L}}_1$ Adaptive Control
• A2: Extracting Cyclic Switchings from DOB-nets for Rejecting Excessive Disturbances
• Bayesian Meta-Learning Through Variational Gaussian Processes
• Robustifying Reinforcement Learning Policies with ${\mathcal{L}}_1$ Adaptive Control