Generalizable Episodic Memory for Deep Reinforcement Learning
arXiv:2103.06469
Abstract
Episodic memory-based methods can rapidly latch onto past successful strategies by a non-parametric memory and improve sample efficiency of traditional reinforcement learning. However, little effort is put into the continuous domain, where a state is never visited twice, and previous episodic methods fail to efficiently aggregate experience across trajectories. To address this problem, we propose Generalizable Episodic Memory (GEM), which effectively organizes the state-action values of episodic memory in a generalizable manner and supports implicit planning on memorized trajectories. GEM utilizes a double estimator to reduce the overestimation bias induced by value propagation in the planning process. Empirical evaluation shows that our method significantly outperforms existing trajectory-based methods on various MuJoCo continuous control tasks. To further show the general applicability, we evaluate our method on Atari games with discrete action space, which also shows a significant improvement over baseline algorithms.
References in corpus (13)
- Continuous control with deep reinforcement learning
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Addressing Function Approximation Error in Actor-Critic Methods
- Dueling Network Architectures for Deep Reinforcement Learning
- QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation
- Model-Free Episodic Control
- Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning
- Self-Imitation Learning
- Deep Reinforcement Learning and the Deadly Triad
- Neural Episodic Control
- Learning to Play in a Day: Faster Deep Reinforcement Learning by Optimality Tightening
- Randomized Ensembled Double Q-Learning: Learning Fast Without a Model
- Convergent Tree Backup and Retrace with Function Approximation