Reset-Free Lifelong Learning with Skill-Space Planning
arXiv:2012.03548
Abstract
The objective of lifelong reinforcement learning (RL) is to optimize agents which can continuously adapt and interact in changing environments. However, current RL approaches fail drastically when environments are non-stationary and interactions are non-episodic. We propose Lifelong Skill Planning (LiSP), an algorithmic framework for non-episodic lifelong RL based on planning in an abstract space of higher-order skills. We learn the skills in an unsupervised manner using intrinsic rewards and plan over the learned skills using a learned dynamics model. Moreover, our framework permits skill discovery even from offline data, thereby reducing the need for excessive real-world interactions. We demonstrate empirically that LiSP successfully enables long-horizon planning and learns agents that can avoid catastrophic failures even in challenging non-stationary and non-episodic environments derived from gridworld and MuJoCo benchmarks.
arXiv commentIn the proceedings of the 7th International Conference on Learning Representations (ICLR), Virtual, April 2021
References in corpus (36)
- Overcoming catastrophic forgetting in neural networks
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
- Conservative Q-Learning for Offline Reinforcement Learning
- Learning Latent Dynamics for Planning from Pixels
- D4RL: Datasets for Deep Data-Driven Reinforcement Learning
- Model-Ensemble Trust-Region Policy Optimization
- Behavior Regularized Offline Reinforcement Learning
- Data-Efficient Hierarchical Reinforcement Learning
- MOPO: Model-based Offline Policy Optimization
- Uncertainty-Aware Reinforcement Learning for Collision Avoidance
- Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
- Variational Intrinsic Control
- Benchmarking Batch Deep Reinforcement Learning Algorithms
- MOReL : Model-Based Offline Reinforcement Learning
- Online Meta-Learning
- Variational Option Discovery Algorithms
- Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic
- Exploring Model-based Planning with Policy Networks
- Dynamics-Aware Unsupervised Discovery of Skills
- Model Predictive Path Integral Control using Covariance Variable Importance Sampling
- Plan Online, Learn Offline: Efficient Learning and Exploration via Model-Based Control
- Why Does Hierarchy (Sometimes) Work So Well in Reinforcement Learning?
- Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL
- The MineRL 2019 Competition on Sample Efficient Reinforcement Learning using Human Priors
- Deep Reinforcement Learning amidst Lifelong Non-Stationarity
- Cautious Adaptation For Reinforcement Learning in Safety-Critical Settings
- Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learning
- Fast Task Inference with Variational Intrinsic Successor Features
- Model-Based Offline Planning
- Ecological Reinforcement Learning
- Emergent Real-World Robotic Skills via Unsupervised Off-Policy Reinforcement Learning
- Adaptive Online Planning for Continual Lifelong Learning
- Non-Stationary Markov Decision Processes, a Worst-Case Approach using Model-Based Reinforcement Learning, Extended version
- Efficient Empowerment Estimation for Unsupervised Stabilization
- Learning Off-Policy with Online Planning