Meta Reinforcement Learning with Distribution of Exploration Parameters Learned by Evolution Strategies

Abstract

In this paper, we propose a novel meta-learning method in a reinforcement learning setting, based on evolution strategies (ES), exploration in parameter space and deterministic policy gradients. ES methods are easy to parallelize, which is desirable for modern training architectures; however, such methods typically require a huge number of samples for effective training. We use deterministic policy gradients during adaptation and other techniques to compensate for the sample-efficiency problem while maintaining the inherent scalability of ES methods. We demonstrate that our method achieves good results compared to gradient-based meta-learning in high-dimensional control tasks in the MuJoCo simulator. In addition, because of gradient-free methods in the meta-training phase, which do not need information about gradients and policies in adaptation training, we predict and confirm our algorithm performs better in tasks that need multi-step adaptation.

References in corpus (6)

• Adam: A Method for Stochastic Optimization
• Continuous control with deep reinforcement learning
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Playing Atari with Deep Reinforcement Learning
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Meta-Learning by the Baldwin Effect