Learning Self-Imitating Diverse Policies

Abstract

The success of popular algorithms for deep reinforcement learning, such as policy-gradients and Q-learning, relies heavily on the availability of an informative reward signal at each timestep of the sequential decision-making process. When rewards are only sparsely available during an episode, or a rewarding feedback is provided only after episode termination, these algorithms perform sub-optimally due to the difficultly in credit assignment. Alternatively, trajectory-based policy optimization methods, such as cross-entropy method and evolution strategies, do not require per-timestep rewards, but have been found to suffer from high sample complexity by completing forgoing the temporal nature of the problem. Improving the efficiency of RL algorithms in real-world problems with sparse or episodic rewards is therefore a pressing need. In this work, we introduce a self-imitation learning algorithm that exploits and explores well in the sparse and episodic reward settings. We view each policy as a state-action visitation distribution and formulate policy optimization as a divergence minimization problem. We show that with Jensen-Shannon divergence, this divergence minimization problem can be reduced into a policy-gradient algorithm with shaped rewards learned from experience replays. Experimental results indicate that our algorithm works comparable to existing algorithms in environments with dense rewards, and significantly better in environments with sparse and episodic rewards. We then discuss limitations of self-imitation learning, and propose to solve them by using Stein variational policy gradient descent with the Jensen-Shannon kernel to learn multiple diverse policies. We demonstrate its effectiveness on a challenging variant of continuous-control MuJoCo locomotion tasks.

arXiv commentICLR 2019

References in corpus (17)

• Continuous control with deep reinforcement learning
• End to End Learning for Self-Driving Cars
• Trust Region Policy Optimization
• Addressing Function Approximation Error in Actor-Critic Methods
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization
• Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
• VIME: Variational Information Maximizing Exploration
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• #Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learning
• Incentivizing Exploration In Reinforcement Learning With Deep Predictive Models
• Stochastic Neural Networks for Hierarchical Reinforcement Learning
• Stein Variational Policy Gradient
• EX2: Exploration with Exemplar Models for Deep Reinforcement Learning
• Neural Symbolic Machines: Learning Semantic Parsers on Freebase with Weak Supervision
• Neural Program Synthesis with Priority Queue Training
• Mutual Alignment Transfer Learning

Cited by in corpus (17)

• Generative Adversarial Self-Imitation Learning
• Active Domain Randomization
• Simplifying Deep Reinforcement Learning via Self-Supervision
• Self-Imitation Learning via Generalized Lower Bound Q-learning
• State-only Imitation with Transition Dynamics Mismatch
• Diversity Actor-Critic: Sample-Aware Entropy Regularization for Sample-Efficient Exploration
• DiversityGAN: Diversity-Aware Vehicle Motion Prediction via Latent Semantic Sampling
• Learning Guidance Rewards with Trajectory-space Smoothing
• Harnessing Distribution Ratio Estimators for Learning Agents with Quality and Diversity
• Amplifying the Imitation Effect for Reinforcement Learning of UCAV's Mission Execution
• Novel Policy Seeking with Constrained Optimization
• Discovering Generalizable Skills via Automated Generation of Diverse Tasks
• Episodic Self-Imitation Learning with Hindsight
• A Closer Look at Advantage-Filtered Behavioral Cloning in High-Noise Datasets
• Nested-Wasserstein Self-Imitation Learning for Sequence Generation
• Self-Imitation Learning for Robot Tasks with Sparse and Delayed Rewards
• Lucid Dreaming for Experience Replay: Refreshing Past States with the Current Policy