Learning Diverse Policies with Soft Self-Generated Guidance

Abstract

Reinforcement learning (RL) with sparse and deceptive rewards is challenging because non-zero rewards are rarely obtained. Hence, the gradient calculated by the agent can be stochastic and without valid information. Recent studies that utilize memory buffers of previous experiences can lead to a more efficient learning process. However, existing methods often require these experiences to be successful and may overly exploit them, which can cause the agent to adopt suboptimal behaviors. This paper develops an approach that uses diverse past trajectories for faster and more efficient online RL, even if these trajectories are suboptimal or not highly rewarded. The proposed algorithm combines a policy improvement step with an additional exploration step using offline demonstration data. The main contribution of this paper is that by regarding diverse past trajectories as guidance, instead of imitating them, our method directs its policy to follow and expand past trajectories while still being able to learn without rewards and approach optimality. Furthermore, a novel diversity measurement is introduced to maintain the team's diversity and regulate exploration. The proposed algorithm is evaluated on discrete and continuous control tasks with sparse and deceptive rewards. Compared with the existing RL methods, the experimental results indicate that our proposed algorithm is significantly better than the baseline methods regarding diverse exploration and avoiding local optima.

23 pages, 19 figures

References in corpus (21)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Addressing Function Approximation Error in Actor-Critic Methods
• Prioritized Experience Replay
• Benchmarking Deep Reinforcement Learning for Continuous Control
• Emergence of Locomotion Behaviours in Rich Environments
• Noisy Networks for Exploration
• VIME: Variational Information Maximizing Exploration
• Parameter Space Noise for Exploration
• Episodic Curiosity through Reachability
• Stochastic Neural Networks for Hierarchical Reinforcement Learning
• Training generative neural networks via Maximum Mean Discrepancy optimization
• Model-Free Episodic Control
• Deep Exploration via Randomized Value Functions
• Self-Imitation Learning
• Improving Exploration in Evolution Strategies for Deep Reinforcement Learning via a Population of Novelty-Seeking Agents
• Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsing
• Generative Adversarial Self-Imitation Learning
• Memory Based Trajectory-conditioned Policies for Learning from Sparse Rewards
• Diversity Actor-Critic: Sample-Aware Entropy Regularization for Sample-Efficient Exploration
• Non-local Policy Optimization via Diversity-regularized Collaborative Exploration