Self-Imitation Learning

Abstract

This paper proposes Self-Imitation Learning (SIL), a simple off-policy actor-critic algorithm that learns to reproduce the agent's past good decisions. This algorithm is designed to verify our hypothesis that exploiting past good experiences can indirectly drive deep exploration. Our empirical results show that SIL significantly improves advantage actor-critic (A2C) on several hard exploration Atari games and is competitive to the state-of-the-art count-based exploration methods. We also show that SIL improves proximal policy optimization (PPO) on MuJoCo tasks.

Cited by in corpus (30)

• Exploration in Deep Reinforcement Learning: A Survey
• Go-Explore: a New Approach for Hard-Exploration Problems
• Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domain
• Transfer Learning for Related Reinforcement Learning Tasks via Image-to-Image Translation
• Generative Adversarial Self-Imitation Learning
• Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation
• Learning to Reach Goals via Iterated Supervised Learning
• Optimistic Reinforcement Learning by Forward Kullback-Leibler Divergence Optimization
• Generalizable Episodic Memory for Deep Reinforcement Learning
• Training Agents using Upside-Down Reinforcement Learning
• Regularized Softmax Deep Multi-Agent $Q$-Learning
• Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills
• Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills
• Sample Efficient Reinforcement Learning through Learning from Demonstrations in Minecraft
• TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations
• HiER: Highlight Experience Replay for Boosting Off-Policy Reinforcement Learning Agents
• Robust Multi-Modal Policies for Industrial Assembly via Reinforcement Learning and Demonstrations: A Large-Scale Study
• Trajectory Optimization for Unknown Constrained Systems using Reinforcement Learning
• Exploring Restart Distributions
• Learning Diverse Policies with Soft Self-Generated Guidance
• Co-Imitation Learning without Expert Demonstration
• Hierarchical Representation Learning for Markov Decision Processes
• GAN-based Intrinsic Exploration For Sample Efficient Reinforcement Learning
• SCAPE: Learning Stiffness Control from Augmented Position Control Experiences
• Self-Imitation Learning for Robot Tasks with Sparse and Delayed Rewards
• DM${}^2$: Decentralized Multi-Agent Reinforcement Learning for Distribution Matching
• Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning
• Reinforcement Learning With Sparse-Executing Actions via Sparsity Regularization
• Learning to Superoptimize Real-world Programs
• Lucid Dreaming for Experience Replay: Refreshing Past States with the Current Policy