Evolutionary Stochastic Policy Distillation

Abstract

Solving the Goal-Conditioned Reward Sparse (GCRS) task is a challenging reinforcement learning problem due to the sparsity of reward signals. In this work, we propose a new formulation of GCRS tasks from the perspective of the drifted random walk on the state space, and design a novel method called Evolutionary Stochastic Policy Distillation (ESPD) to solve them based on the insight of reducing the First Hitting Time of the stochastic process. As a self-imitate approach, ESPD enables a target policy to learn from a series of its stochastic variants through the technique of policy distillation (PD). The learning mechanism of ESPD can be considered as an Evolution Strategy (ES) that applies perturbations upon policy directly on the action space, with a SELECT function to check the superiority of stochastic variants and then use PD to update the policy. The experiments based on the MuJoCo robotics control suite show the high learning efficiency of the proposed method.

References in corpus (17)

• Continuous control with deep reinforcement learning
• Addressing Function Approximation Error in Actor-Critic Methods
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Parameter Space Noise for Exploration
• Hindsight Experience Replay
• Deep Q-learning from Demonstrations
• Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research
• Reinforcement Learning from Imperfect Demonstrations
• Kickstarting Deep Reinforcement Learning
• Distilling Policy Distillation
• Learning to Reach Goals via Iterated Supervised Learning
• Training Agents using Upside-Down Reinforcement Learning
• Reinforcement Learning Upside Down: Don't Predict Rewards -- Just Map Them to Actions
• Multi-task Learning for Continuous Control
• Policy Continuation with Hindsight Inverse Dynamics
• Policy Search by Target Distribution Learning for Continuous Control
• Importance Weighted Evolution Strategies