Experience Replay Optimization

Abstract

Experience replay enables reinforcement learning agents to memorize and reuse past experiences, just as humans replay memories for the situation at hand. Contemporary off-policy algorithms either replay past experiences uniformly or utilize a rule-based replay strategy, which may be sub-optimal. In this work, we consider learning a replay policy to optimize the cumulative reward. Replay learning is challenging because the replay memory is noisy and large, and the cumulative reward is unstable. To address these issues, we propose a novel experience replay optimization (ERO) framework which alternately updates two policies: the agent policy, and the replay policy. The agent is updated to maximize the cumulative reward based on the replayed data, while the replay policy is updated to provide the agent with the most useful experiences. The conducted experiments on various continuous control tasks demonstrate the effectiveness of ERO, empirically showing promise in experience replay learning to improve the performance of off-policy reinforcement learning algorithms.

References in corpus (7)

• Playing Atari with Deep Reinforcement Learning
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Hindsight Experience Replay
• A Deeper Look at Experience Replay
• Learning to Teach with Dynamic Loss Functions
• Learning to Explore with Meta-Policy Gradient
• The Effects of Memory Replay in Reinforcement Learning

Cited by in corpus (7)

• Deep Reinforcement Learning in Computer Vision: A Comprehensive Survey
• Explanation-Aware Experience Replay in Rule-Dense Environments
• Meta-AAD: Active Anomaly Detection with Deep Reinforcement Learning
• Policy-GNN: Aggregation Optimization for Graph Neural Networks
• Revisiting Prioritized Experience Replay: A Value Perspective
• Locality-Sensitive Experience Replay for Online Recommendation
• Model-Augmented Q-learning