Parallelized Reverse Curriculum Generation

Abstract

For reinforcement learning (RL), it is challenging for an agent to master a task that requires a specific series of actions due to sparse rewards. To solve this problem, reverse curriculum generation (RCG) provides a reverse expansion approach that automatically generates a curriculum for the agent to learn. More specifically, RCG adapts the initial state distribution from the neighborhood of a goal to a distance as training proceeds. However, the initial state distribution generated for each iteration might be biased, thus making the policy overfit or slowing down the reverse expansion rate. While training RCG for actor-critic (AC) based RL algorithms, this poor generalization and slow convergence might be induced by the tight coupling between an AC pair. Therefore, we propose a parallelized approach that simultaneously trains multiple AC pairs and periodically exchanges their critics. We empirically demonstrate that this proposed approach can improve RCG in performance and convergence, and it can also be applied to other AC based RL algorithms with adapted initial state distribution.

References in corpus (14)

• Continuous control with deep reinforcement learning
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• NIPS 2016 Tutorial: Generative Adversarial Networks
• Emergence of Locomotion Behaviours in Rich Environments
• Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards
• Massively Parallel Methods for Deep Reinforcement Learning
• Hindsight Experience Replay
• Learning by Playing - Solving Sparse Reward Tasks from Scratch
• Reverse Curriculum Generation for Reinforcement Learning
• Data-efficient Deep Reinforcement Learning for Dexterous Manipulation
• Connecting Generative Adversarial Networks and Actor-Critic Methods
• Distributed Deep Q-Learning
• Generative Adversarial Parallelization
• Evolutionary Generative Adversarial Networks