OER: Offline Experience Replay for Continual Offline Reinforcement Learning

Abstract

The capability of continuously learning new skills via a sequence of pre-collected offline datasets is desired for an agent. However, consecutively learning a sequence of offline tasks likely leads to the catastrophic forgetting issue under resource-limited scenarios. In this paper, we formulate a new setting, continual offline reinforcement learning (CORL), where an agent learns a sequence of offline reinforcement learning tasks and pursues good performance on all learned tasks with a small replay buffer without exploring any of the environments of all the sequential tasks. For consistently learning on all sequential tasks, an agent requires acquiring new knowledge and meanwhile preserving old knowledge in an offline manner. To this end, we introduced continual learning algorithms and experimentally found experience replay (ER) to be the most suitable algorithm for the CORL problem. However, we observe that introducing ER into CORL encounters a new distribution shift problem: the mismatch between the experiences in the replay buffer and trajectories from the learned policy. To address such an issue, we propose a new model-based experience selection (MBES) scheme to build the replay buffer, where a transition model is learned to approximate the state distribution. This model is used to bridge the distribution bias between the replay buffer and the learned model by filtering the data from offline data that most closely resembles the learned model for storage. Moreover, in order to enhance the ability on learning new tasks, we retrofit the experience replay method with a new dual behavior cloning (DBC) architecture to avoid the disturbance of behavior-cloning loss on the Q-learning process. In general, we call our algorithm offline experience replay (OER). Extensive experiments demonstrate that our OER method outperforms SOTA baselines in widely-used Mujoco environments.

9 pages, 4 figures

References in corpus (38)

• Adam: A Method for Stochastic Optimization
• Overcoming catastrophic forgetting in neural networks
• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Soft Actor-Critic Algorithms and Applications
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Efficient Lifelong Learning with A-GEM
• Conservative Q-Learning for Offline Reinforcement Learning
• Experience Replay for Continual Learning
• Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction
• D4RL: Datasets for Deep Data-Driven Reinforcement Learning
• Off-Policy Deep Reinforcement Learning without Exploration
• MOPO: Model-based Offline Policy Optimization
• Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning
• A Minimalist Approach to Offline Reinforcement Learning
• MOReL : Model-Based Offline Reinforcement Learning
• Offline Reinforcement Learning with Implicit Q-Learning
• An Optimistic Perspective on Offline Reinforcement Learning
• Critic Regularized Regression
• Connecting Generative Adversarial Networks and Actor-Critic Methods
• Dark Experience for General Continual Learning: a Strong, Simple Baseline
• AWAC: Accelerating Online Reinforcement Learning with Offline Datasets
• COMBO: Conservative Offline Model-Based Policy Optimization
• d3rlpy: An Offline Deep Reinforcement Learning Library
• Online Coreset Selection for Rehearsal-based Continual Learning
• Gradient Projection Memory for Continual Learning
• Mildly Conservative Q-Learning for Offline Reinforcement Learning
• Conservative Offline Distributional Reinforcement Learning
• Offline Meta-Reinforcement Learning with Advantage Weighting
• Lifelong Policy Gradient Learning of Factored Policies for Faster Training Without Forgetting
• Policy Consolidation for Continual Reinforcement Learning
• DARA: Dynamics-Aware Reward Augmentation in Offline Reinforcement Learning
• Behavior Proximal Policy Optimization
• Disentangling Transfer in Continual Reinforcement Learning
• Offline Reinforcement Learning with Pseudometric Learning
• Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learning
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning