Remember and Forget for Experience Replay

Abstract

Experience replay (ER) is a fundamental component of off-policy deep reinforcement learning (RL). ER recalls experiences from past iterations to compute gradient estimates for the current policy, increasing data-efficiency. However, the accuracy of such updates may deteriorate when the policy diverges from past behaviors and can undermine the performance of ER. Many algorithms mitigate this issue by tuning hyper-parameters to slow down policy changes. An alternative is to actively enforce the similarity between policy and the experiences in the replay memory. We introduce Remember and Forget Experience Replay (ReF-ER), a novel method that can enhance RL algorithms with parameterized policies. ReF-ER (1) skips gradients computed from experiences that are too unlikely with the current policy and (2) regulates policy changes within a trust region of the replayed behaviors. We couple ReF-ER with Q-learning, deterministic policy gradient and off-policy gradient methods. We find that ReF-ER consistently improves the performance of continuous-action, off-policy RL on fully observable benchmarks and partially observable flow control problems.

References in corpus (11)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Continuous Deep Q-Learning with Model-based Acceleration
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• Flow Navigation by Smart Microswimmers via Reinforcement Learning
• Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
• Synchronised Swimming of Two Fish
• The Mirage of Action-Dependent Baselines in Reinforcement Learning
• Recall Traces: Backtracking Models for Efficient Reinforcement Learning

Cited by in corpus (6)

• A Theoretical Analysis of Deep Q-Learning
• Log-law recovery through reinforcement-learning wall model for large-eddy simulation
• Policy Search by Target Distribution Learning for Continuous Control
• Deep-Reinforcement-Learning for Gliding and Perching Bodies
• Improved Memories Learning
• Locality-Sensitive Experience Replay for Online Recommendation