Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL

Abstract

Humans and animals can learn complex predictive models that allow them to accurately and reliably reason about real-world phenomena, and they can adapt such models extremely quickly in the face of unexpected changes. Deep neural network models allow us to represent very complex functions, but lack this capacity for rapid online adaptation. The goal in this paper is to develop a method for continual online learning from an incoming stream of data, using deep neural network models. We formulate an online learning procedure that uses stochastic gradient descent to update model parameters, and an expectation maximization algorithm with a Chinese restaurant process prior to develop and maintain a mixture of models to handle non-stationary task distributions. This allows for all models to be adapted as necessary, with new models instantiated for task changes and old models recalled when previously seen tasks are encountered again. Furthermore, we observe that meta-learning can be used to meta-train a model such that this direct online adaptation with SGD is effective, which is otherwise not the case for large function approximators. In this work, we apply our meta-learning for online learning (MOLe) approach to model-based reinforcement learning, where adapting the predictive model is critical for control; we demonstrate that MOLe outperforms alternative prior methods, and enables effective continuous adaptation in non-stationary task distributions such as varying terrains, motor failures, and unexpected disturbances.

Project website: https://sites.google.com/berkeley.edu/onlineviameta

References in corpus (12)

• Overcoming catastrophic forgetting in neural networks
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Theoretical Models of Learning to Learn
• Continual Learning Through Synaptic Intelligence
• Learning to reinforcement learn
• Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environments
• Bayesian Recurrent Neural Networks
• Streaming Variational Bayes
• Multiplicative LSTM for sequence modelling
• Model Predictive Path Integral Control using Covariance Variable Importance Sampling
• Dynamic Evaluation of Neural Sequence Models
• Been There, Done That: Meta-Learning with Episodic Recall

Cited by in corpus (36)

• Generalizing from a Few Examples: A Survey on Few-Shot Learning
• A Survey of Zero-shot Generalisation in Deep Reinforcement Learning
• Learning to Generalize from Sparse and Underspecified Rewards
• Lifelong Incremental Reinforcement Learning with Online Bayesian Inference
• Learning Compositional Koopman Operators for Model-Based Control
• An empirical investigation of the challenges of real-world reinforcement learning
• Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learning
• A Comprehensive Overview and Survey of Recent Advances in Meta-Learning
• La-MAML: Look-ahead Meta Learning for Continual Learning
• TaskNorm: Rethinking Batch Normalization for Meta-Learning
• Meta-Learning Representations for Continual Learning
• Task Agnostic Continual Learning Using Online Variational Bayes with Fixed-Point Updates
• Model-based Adversarial Meta-Reinforcement Learning
• Lifelong Policy Gradient Learning of Factored Policies for Faster Training Without Forgetting
• Reset-Free Lifelong Learning with Skill-Space Planning
• Lifelong Learning of Compositional Structures
• Weighted Meta-Learning
• AOL: Adaptive Online Learning for Human Trajectory Prediction in Dynamic Video Scenes
• Augmented World Models Facilitate Zero-Shot Dynamics Generalization From a Single Offline Environment
• Continuous Meta-Learning without Tasks
• Online Structured Meta-learning
• Adaptive Online Planning for Continual Lifelong Learning
• Nonstationary Nonparametric Online Learning: Balancing Dynamic Regret and Model Parsimony
• Learning State-Dependent Losses for Inverse Dynamics Learning
• Continual Model-Based Reinforcement Learning with Hypernetworks
• Training for the Future: A Simple Gradient Interpolation Loss to Generalize Along Time
• MLDemon: Deployment Monitoring for Machine Learning Systems
• Meta-Model-Based Meta-Policy Optimization
• Meta-Reinforcement Learning in Broad and Non-Parametric Environments
• Task-Agnostic Online Reinforcement Learning with an Infinite Mixture of Gaussian Processes
• PerSim: Data-Efficient Offline Reinforcement Learning with Heterogeneous Agents via Personalized Simulators
• Online Meta Adaptation for Variable-Rate Learned Image Compression
• Robustifying Reinforcement Learning Policies with ${\mathcal{L}}_1$ Adaptive Control
• Online Continual Learning in Image Classification: An Empirical Survey
• Characterizing Policy Divergence for Personalized Meta-Reinforcement Learning
• A method of supervised learning from conflicting data with hidden contexts