Dynamic Experience Replay
arXiv:2003.02372
Abstract
We present a novel technique called Dynamic Experience Replay (DER) that allows Reinforcement Learning (RL) algorithms to use experience replay samples not only from human demonstrations but also successful transitions generated by RL agents during training and therefore improve training efficiency. It can be combined with an arbitrary off-policy RL algorithm, such as DDPG or DQN, and their distributed versions. We build upon Ape-X DDPG and demonstrate our approach on robotic tight-fitting joint assembly tasks, based on force/torque and Cartesian pose observations. In particular, we run experiments on two different tasks: peg-in-hole and lap-joint. In each case, we compare different replay buffer structures and how DER affects them. Our ablation studies show that Dynamic Experience Replay is a crucial ingredient that either largely shortens the training time in these challenging environments or solves the tasks that the vanilla Ape-X DDPG cannot solve. We also show that our policies learned purely in simulation can be deployed successfully on the real robot. The video presenting our experiments is available at https://sites.google.com/site/dynamicexperiencereplay
arXiv comment10 pages, 5 figures, presented at 2019 Conference on Robot Learning (CoRL)
References in corpus (12)
- Adam: A Method for Stochastic Optimization
- SMOTE: Synthetic Minority Over-sampling Technique
- Continuous control with deep reinforcement learning
- Trust Region Policy Optimization
- Prioritized Experience Replay
- Emergence of Locomotion Behaviours in Rich Environments
- Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
- Distributed Prioritized Experience Replay
- Hindsight Experience Replay
- Distributed Distributional Deterministic Policy Gradients
- Reinforcement Learning on Variable Impedance Controller for High-Precision Robotic Assembly
- A Learning Framework for High Precision Industrial Assembly