Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning

Abstract

Although reinforcement learning methods can achieve impressive results in simulation, the real world presents two major challenges: generating samples is exceedingly expensive, and unexpected perturbations or unseen situations cause proficient but specialized policies to fail at test time. Given that it is impractical to train separate policies to accommodate all situations the agent may see in the real world, this work proposes to learn how to quickly and effectively adapt online to new tasks. To enable sample-efficient learning, we consider learning online adaptation in the context of model-based reinforcement learning. Our approach uses meta-learning to train a dynamics model prior such that, when combined with recent data, this prior can be rapidly adapted to the local context. Our experiments demonstrate online adaptation for continuous control tasks on both simulated and real-world agents. We first show simulated agents adapting their behavior online to novel terrains, crippled body parts, and highly-dynamic environments. We also illustrate the importance of incorporating online adaptation into autonomous agents that operate in the real world by applying our method to a real dynamic legged millirobot. We demonstrate the agent's learned ability to quickly adapt online to a missing leg, adjust to novel terrains and slopes, account for miscalibration or errors in pose estimation, and compensate for pulling payloads.

First 2 authors contributed equally. Website: https://sites.google.com/berkeley.edu/metaadaptivecontrol

References in corpus (16)

• Continuous control with deep reinforcement learning
• Trust Region Policy Optimization
• A Simple Neural Attentive Meta-Learner
• Learning to reinforcement learn
• Continuous Deep Q-Learning with Model-based Acceleration
• One-shot Learning with Memory-Augmented Neural Networks
• Model-Ensemble Trust-Region Policy Optimization
• Learning to Optimize
• Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments
• Bayesian Recurrent Neural Networks
• Meta Reinforcement Learning with Latent Variable Gaussian Processes
• Learning to Learn: Meta-Critic Networks for Sample Efficient Learning
• Meta-Learning and Universality: Deep Representations and Gradient Descent can Approximate any Learning Algorithm
• Multiplicative LSTM for sequence modelling
• Model Predictive Path Integral Control using Covariance Variable Importance Sampling
• Learning Feedback Terms for Reactive Planning and Control

Cited by in corpus (13)

• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• learn2learn: A Library for Meta-Learning Research
• A Practical Guide to Multi-Objective Reinforcement Learning and Planning
• BADGER: Learning to (Learn [Learning Algorithms] through Multi-Agent Communication)
• MULTIPOLAR: Multi-Source Policy Aggregation for Transfer Reinforcement Learning between Diverse Environmental Dynamics
• Trajectory-wise Multiple Choice Learning for Dynamics Generalization in Reinforcement Learning
• Unsupervised Meta-Testing with Conditional Neural Processes for Hybrid Meta-Reinforcement Learning
• An empirical study of pretrained representations for few-shot classification
• Meta Learning MPC using Finite-Dimensional Gaussian Process Approximations
• A Simple Approach to Continual Learning by Transferring Skill Parameters
• Meta-Learning with Versatile Loss Geometries for Fast Adaptation Using Mirror Descent
• Performance-Weighed Policy Sampling for Meta-Reinforcement Learning