Kickstarting Deep Reinforcement Learning

Abstract

We present a method for using previously-trained 'teacher' agents to kickstart the training of a new 'student' agent. To this end, we leverage ideas from policy distillation and population based training. Our method places no constraints on the architecture of the teacher or student agents, and it regulates itself to allow the students to surpass their teachers in performance. We show that, on a challenging and computationally-intensive multi-task benchmark (DMLab-30), kickstarted training improves the data efficiency of new agents, making it significantly easier to iterate on their design. We also show that the same kickstarting pipeline can allow a single student agent to leverage multiple 'expert' teachers which specialize on individual tasks. In this setting kickstarting yields surprisingly large gains, with the kickstarted agent matching the performance of an agent trained from scratch in almost 10x fewer steps, and surpassing its final performance by 42 percent. Kickstarting is conceptually simple and can easily be incorporated into reinforcement learning experiments.

References in corpus (8)

• Distilling the Knowledge in a Neural Network
• Do Deep Nets Really Need to be Deep?
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Deep Q-learning from Demonstrations
• Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
• Distral: Robust Multitask Reinforcement Learning
• Psychlab: A Psychology Laboratory for Deep Reinforcement Learning Agents
• Training Recurrent Neural Networks by Diffusion

Cited by in corpus (30)

• Transfer Learning in Deep Reinforcement Learning: A Survey
• Adapting Auxiliary Losses Using Gradient Similarity
• Open-Ended Learning Leads to Generally Capable Agents
• Discrete and Continuous Action Representation for Practical RL in Video Games
• V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control
• Distilling Policy Distillation
• Multi-task Deep Reinforcement Learning with PopArt
• Backplay: "Man muss immer umkehren"
• Exploiting Hierarchy for Learning and Transfer in KL-regularized RL
• Open Problems in Cooperative AI
• Provably Efficient Online Hyperparameter Optimization with Population-Based Bandits
• Self-Attentional Credit Assignment for Transfer in Reinforcement Learning
• Deep neuroethology of a virtual rodent
• Transient Non-Stationarity and Generalisation in Deep Reinforcement Learning
• ZPD Teaching Strategies for Deep Reinforcement Learning from Demonstrations
• Lifelong Robotic Reinforcement Learning by Retaining Experiences
• SmartChoices: Hybridizing Programming and Machine Learning
• Efficient Deep Reinforcement Learning via Adaptive Policy Transfer
• Mutual Information Based Knowledge Transfer Under State-Action Dimension Mismatch
• REPAINT: Knowledge Transfer in Deep Reinforcement Learning
• Privacy-Preserving Kickstarting Deep Reinforcement Learning with Privacy-Aware Learners
• Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search
• Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRL
• Substitute Teacher Networks: Learning with Almost No Supervision
• Scaling shared model governance via model splitting
• On Lottery Tickets and Minimal Task Representations in Deep Reinforcement Learning
• Designing Interpretable Approximations to Deep Reinforcement Learning
• Don't Forget Your Teacher: A Corrective Reinforcement Learning Framework
• Evolutionary Stochastic Policy Distillation
• DCUR: Data Curriculum for Teaching via Samples with Reinforcement Learning