Distilling Policy Distillation

Abstract

The transfer of knowledge from one policy to another is an important tool in Deep Reinforcement Learning. This process, referred to as distillation, has been used to great success, for example, by enhancing the optimisation of agents, leading to stronger performance faster, on harder domains [26, 32, 5, 8]. Despite the widespread use and conceptual simplicity of distillation, many different formulations are used in practice, and the subtle variations between them can often drastically change the performance and the resulting objective that is being optimised. In this work, we rigorously explore the entire landscape of policy distillation, comparing the motivations and strengths of each variant through theoretical and empirical analysis. Our results point to three distillation techniques, that are preferred depending on specifics of the task. Specifically a newly proposed expected entropy regularised distillation allows for quicker learning in a wide range of situations, while still guaranteeing convergence.

Accepted at AISTATS 2019

References in corpus (14)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Trust Region Policy Optimization
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
• Equivalence Between Policy Gradients and Soft Q-Learning
• Gradient Estimation Using Stochastic Computation Graphs
• Deep Mutual Learning
• Kickstarting Deep Reinforcement Learning
• Collaborative Deep Reinforcement Learning
• Divide-and-Conquer Reinforcement Learning
• Multi-task Learning for Continuous Control

Cited by in corpus (14)

• Solving Rubik's Cube with a Robot Hand
• Supervised Learning Achieves Human-Level Performance in MOBA Games: A Case Study of Honor of Kings
• Discrete and Continuous Action Representation for Practical RL in Video Games
• Knowledge Transfer in Multi-Task Deep Reinforcement Learning for Continuous Control
• SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policies
• Human-Inspired Multi-Agent Navigation using Knowledge Distillation
• Dual Policy Distillation
• Multi-task Batch Reinforcement Learning with Metric Learning
• Transient Non-Stationarity and Generalisation in Deep Reinforcement Learning
• Mutual Information Based Knowledge Transfer Under State-Action Dimension Mismatch
• Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search
• Learning Efficient Multi-Agent Cooperative Visual Exploration
• Evolutionary Stochastic Policy Distillation
• Trying AGAIN instead of Trying Longer: Prior Learning for Automatic Curriculum Learning