Self-Imitation Learning for Robot Tasks with Sparse and Delayed Rewards

Abstract

The application of reinforcement learning (RL) in robotic control is still limited in the environments with sparse and delayed rewards. In this paper, we propose a practical self-imitation learning method named Self-Imitation Learning with Constant Reward (SILCR). Instead of requiring hand-defined immediate rewards from environments, our method assigns the immediate rewards at each timestep with constant values according to their final episodic rewards. In this way, even if the dense rewards from environments are unavailable, every action taken by the agents would be guided properly. We demonstrate the effectiveness of our method in some challenging continuous robotics control tasks in MuJoCo simulation and the results show that our method significantly outperforms the alternative methods in tasks with sparse and delayed rewards. Even compared with alternatives with dense rewards available, our method achieves competitive performance. The ablation experiments also show the stability and reproducibility of our method.

6 pages, 5 figures

References in corpus (14)

• Learning agile and dynamic motor skills for legged robots
• Dota 2 with Large Scale Deep Reinforcement Learning
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Distributed Prioritized Experience Replay
• Generative Adversarial Imitation Learning
• Deep Q-learning from Demonstrations
• Exploration by Random Network Distillation
• Model-Free Episodic Control
• Self-Imitation Learning
• Neural Episodic Control
• SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards
• Learning Self-Imitating Diverse Policies
• Soft Hindsight Experience Replay
• Pretrain Soft Q-Learning with Imperfect Demonstrations