Mix&Match - Agent Curricula for Reinforcement Learning

Abstract

We introduce Mix&Match (M&M) - a training framework designed to facilitate rapid and effective learning in RL agents, especially those that would be too slow or too challenging to train otherwise. The key innovation is a procedure that allows us to automatically form a curriculum over agents. Through such a curriculum we can progressively train more complex agents by, effectively, bootstrapping from solutions found by simpler agents. In contradistinction to typical curriculum learning approaches, we do not gradually modify the tasks or environments presented, but instead use a process to gradually alter how the policy is represented internally. We show the broad applicability of our method by demonstrating significant performance gains in three different experimental setups: (1) We train an agent able to control more than 700 actions in a challenging 3D first-person task; using our method to progress through an action-space curriculum we achieve both faster training and better final performance than one obtains using traditional methods. (2) We further show that M&M can be used successfully to progress through a curriculum of architectural variants defining an agents internal state. (3) Finally, we illustrate how a variant of our method can be used to improve agent performance in a multitask setting.

arXiv commentICML 2018

References in corpus (8)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Do Deep Nets Really Need to be Deep?
• Emergence of Locomotion Behaviours in Rich Environments
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Learning to Execute
• Distral: Robust Multitask Reinforcement Learning
• Deep Mutual Learning

Cited by in corpus (11)

• Dota 2 with Large Scale Deep Reinforcement Learning
• Adapting Behaviour for Learning Progress
• Mutual Information Based Knowledge Transfer Under State-Action Dimension Mismatch
• Skill Transfer in Deep Reinforcement Learning under Morphological Heterogeneity
• PoBRL: Optimizing Multi-Document Summarization by Blending Reinforcement Learning Policies
• Curriculum Design for Teaching via Demonstrations: Theory and Applications
• Adaptive Procedural Task Generation for Hard-Exploration Problems
• LISPR: An Options Framework for Policy Reuse with Reinforcement Learning
• Periodic Intra-Ensemble Knowledge Distillation for Reinforcement Learning
• Growing Action Spaces
• Maximum Entropy Reinforcement Learning with Mixture Policies