Lucid Dreaming for Experience Replay: Refreshing Past States with the Current Policy

Abstract

Experience replay (ER) improves the data efficiency of off-policy reinforcement learning (RL) algorithms by allowing an agent to store and reuse its past experiences in a replay buffer. While many techniques have been proposed to enhance ER by biasing how experiences are sampled from the buffer, thus far they have not considered strategies for refreshing experiences inside the buffer. In this work, we introduce Lucid Dreaming for Experience Replay (LiDER), a conceptually new framework that allows replay experiences to be refreshed by leveraging the agent's current policy. LiDER consists of three steps: First, LiDER moves an agent back to a past state. Second, from that state, LiDER then lets the agent execute a sequence of actions by following its current policy -- as if the agent were "dreaming" about the past and can try out different behaviors to encounter new experiences in the dream. Third, LiDER stores and reuses the new experience if it turned out better than what the agent previously experienced, i.e., to refresh its memories. LiDER is designed to be easily incorporated into off-policy, multi-worker RL algorithms that use ER; we present in this work a case study of applying LiDER to an actor-critic based algorithm. Results show LiDER consistently improves performance over the baseline in six Atari 2600 games. Our open-source implementation of LiDER and the data used to generate all plots in this work are available at github.com/duyunshu/lucid-dreaming-for-exp-replay.

arXiv comment29 pages (with appendices), 8 figures, preprint

References in corpus (21)

• Continuous control with deep reinforcement learning
• Distributed Prioritized Experience Replay
• Deep Q-learning from Demonstrations
• Sample Efficient Actor-Critic with Experience Replay
• Go-Explore: a New Approach for Hard-Exploration Problems
• A Deeper Look at Experience Replay
• Self-Imitation Learning
• Reverse Curriculum Generation for Reinforcement Learning
• Observe and Look Further: Achieving Consistent Performance on Atari
• Revisiting Fundamentals of Experience Replay
• Learning Montezuma's Revenge from a Single Demonstration
• Playing Atari Games with Deep Reinforcement Learning and Human Checkpoint Replay
• Learning to Play in a Day: Faster Deep Reinforcement Learning by Optimality Tightening
• The Reactor: A fast and sample-efficient Actor-Critic agent for Reinforcement Learning
• Learning Self-Imitating Diverse Policies
• Backplay: "Man muss immer umkehren"
• ACTRCE: Augmenting Experience via Teacher's Advice For Multi-Goal Reinforcement Learning
• Experience Replay with Likelihood-free Importance Weights
• Memory Based Trajectory-conditioned Policies for Learning from Sparse Rewards
• Exploring Restart Distributions
• Jointly Pre-training with Supervised, Autoencoder, and Value Losses for Deep Reinforcement Learning