Adaptive Online Planning for Continual Lifelong Learning
arXiv:1912.01188
Abstract
We study learning control in an online reset-free lifelong learning scenario, where mistakes can compound catastrophically into the future and the underlying dynamics of the environment may change. Traditional model-free policy learning methods have achieved successes in difficult tasks due to their broad flexibility, but struggle in this setting, as they can activate failure modes early in their lifetimes which are difficult to recover from and face performance degradation as dynamics change. On the other hand, model-based planning methods learn and adapt quickly, but require prohibitive levels of computational resources. We present a new algorithm, Adaptive Online Planning (AOP), that achieves strong performance in this setting by combining model-based planning with model-free learning. By approximating the uncertainty of the model-free components and the planner performance, AOP is able to call upon more extensive planning only when necessary, leading to reduced computation times, while still gracefully adapting behaviors in the face of unpredictable changes in the world -- even when traditional RL fails.
arXiv commentOriginally published in NeurIPS Deep RL 2019
References in corpus (19)
- Overcoming catastrophic forgetting in neural networks
- Playing Atari with Deep Reinforcement Learning
- Addressing Function Approximation Error in Actor-Critic Methods
- An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks
- Three scenarios for continual learning
- Deep Exploration via Bootstrapped DQN
- Experience Replay for Continual Learning
- Off-Policy Deep Reinforcement Learning without Exploration
- Model-Ensemble Trust-Region Policy Optimization
- Thinking Fast and Slow with Deep Learning and Tree Search
- Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
- When to Trust Your Model: Model-Based Policy Optimization
- Model-Based Reinforcement Learning via Meta-Policy Optimization
- Reinforcement and Imitation Learning for Diverse Visuomotor Skills
- Exploring Model-based Planning with Policy Networks
- Model Predictive Path Integral Control using Covariance Variable Importance Sampling
- Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control
- Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL
- Learning to Search with MCTSnets