Learning Memory-Dependent Continuous Control from Demonstrations

Abstract

Efficient exploration has presented a long-standing challenge in reinforcement learning, especially when rewards are sparse. A developmental system can overcome this difficulty by learning from both demonstrations and self-exploration. However, existing methods are not applicable to most real-world robotic controlling problems because they assume that environments follow Markov decision processes (MDP); thus, they do not extend to partially observable environments where historical observations are necessary for decision making. This paper builds on the idea of replaying demonstrations for memory-dependent continuous control, by proposing a novel algorithm, Recurrent Actor-Critic with Demonstration and Experience Replay (READER). Experiments involving several memory-crucial continuous control tasks reveal significantly reduce interactions with the environment using our method with a reasonably small number of demonstration samples. The algorithm also shows better sample efficiency and learning capabilities than a baseline reinforcement learning algorithm for memory-based control from demonstrations.

arXiv comment10 pages, 6 figures

References in corpus (11)

• Playing Atari with Deep Reinforcement Learning
• On the difficulty of training Recurrent Neural Networks
• Trust Region Policy Optimization
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Recurrent World Models Facilitate Policy Evolution
• Deep Q-learning from Demonstrations
• FeUdal Networks for Hierarchical Reinforcement Learning
• Memory-based control with recurrent neural networks
• Accelerating Reinforcement Learning through Implicit Imitation
• Q($\lambda$) with Off-Policy Corrections
• Self-organization of action hierarchy and compositionality by reinforcement learning with recurrent neural networks