Prioritized Sequence Experience Replay

Abstract

Experience replay is widely used in deep reinforcement learning algorithms and allows agents to remember and learn from experiences from the past. In an effort to learn more efficiently, researchers proposed prioritized experience replay (PER) which samples important transitions more frequently. In this paper, we propose Prioritized Sequence Experience Replay (PSER) a framework for prioritizing sequences of experience in an attempt to both learn more efficiently and to obtain better performance. We compare the performance of PER and PSER sampling techniques in a tabular Q-learning environment and in DQN on the Atari 2600 benchmark. We prove theoretically that PSER is guaranteed to converge faster than PER and empirically show PSER substantially improves upon PER.

18 pages

References in corpus (11)

• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Dueling Network Architectures for Deep Reinforcement Learning
• Noisy Networks for Exploration
• Distributed Prioritized Experience Replay
• Rainbow: Combining Improvements in Deep Reinforcement Learning
• Sample Efficient Actor-Critic with Experience Replay
• A Deeper Look at Experience Replay
• Dopamine: A Research Framework for Deep Reinforcement Learning
• Sample-Efficient Deep Reinforcement Learning via Episodic Backward Update
• Experience Replay Using Transition Sequences

Cited by in corpus (7)

• An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replay
• Continuous Control for Searching and Planning with a Learned Model
• Learning to Sample with Local and Global Contexts in Experience Replay Buffer
• Improved Soft Actor-Critic: Mixing Prioritized Off-Policy Samples with On-Policy Experience
• State-Aware Variational Thompson Sampling for Deep Q-Networks
• Model-Augmented Q-learning
• Adaptive Experience Selection for Policy Gradient