Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning

Abstract

The ability to act in multiple environments and transfer previous knowledge to new situations can be considered a critical aspect of any intelligent agent. Towards this goal, we define a novel method of multitask and transfer learning that enables an autonomous agent to learn how to behave in multiple tasks simultaneously, and then generalize its knowledge to new domains. This method, termed "Actor-Mimic", exploits the use of deep reinforcement learning and model compression techniques to train a single policy network that learns how to act in a set of distinct tasks by using the guidance of several expert teachers. We then show that the representations learnt by the deep policy network are capable of generalizing to new tasks with no prior expert guidance, speeding up learning in novel environments. Although our method can in general be applied to a wide range of problems, we use Atari games as a testing environment to demonstrate these methods.

Accepted as a conference paper at ICLR 2016

References in corpus (1)

• Distilling the Knowledge in a Neural Network

Cited by in corpus (90)

• Overcoming catastrophic forgetting in neural networks
• An Introduction to Deep Reinforcement Learning
• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• Multi-Task Learning with Deep Neural Networks: A Survey
• Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning
• Fully Decentralized Multi-Agent Reinforcement Learning with Networked Agents
• Transfer Learning in Deep Reinforcement Learning: A Survey
• Towards Deep Symbolic Reinforcement Learning
• A Deep Hierarchical Approach to Lifelong Learning in Minecraft
• Multi-Agent Reinforcement Learning via Double Averaging Primal-Dual Optimization
• Gradient Surgery for Multi-Task Learning
• Generalization and Regularization in DQN
• Gotta Learn Fast: A New Benchmark for Generalization in RL
• Sharing Knowledge in Multi-Task Deep Reinforcement Learning
• Dynamic Weights in Multi-Objective Deep Reinforcement Learning
• Multi-Task Reinforcement Learning with Soft Modularization
• Improving Generalization in Meta Reinforcement Learning using Learned Objectives
• Graying the black box: Understanding DQNs
• Uses and Abuses of the Cross-Entropy Loss: Case Studies in Modern Deep Learning
• Kickstarting Deep Reinforcement Learning
• Distilling Policy Distillation
• Discrete and Continuous Action Representation for Practical RL in Video Games
• 12-in-1: Multi-Task Vision and Language Representation Learning
• MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale
• UPDeT: Universal Multi-agent Reinforcement Learning via Policy Decoupling with Transformers
• Rewriting History with Inverse RL: Hindsight Inference for Policy Improvement
• Multi-task Deep Reinforcement Learning with PopArt
• Pre-training with Non-expert Human Demonstration for Deep Reinforcement Learning
• Active Long Term Memory Networks
• Towards Effective Low-bitwidth Convolutional Neural Networks
• Progressive Reinforcement Learning with Distillation for Multi-Skilled Motion Control
• Multi-Task Reinforcement Learning with Context-based Representations
• Generalized Hindsight for Reinforcement Learning
• Collaborative Deep Reinforcement Learning
• Benchmark Environments for Multitask Learning in Continuous Domains
• Universally Slimmable Networks and Improved Training Techniques
• Meta-learning curiosity algorithms
• Fast Adaptation via Policy-Dynamics Value Functions
• Incremental Sequence Learning
• Context-Aware Policy Reuse
• Symbolic Network: Generalized Neural Policies for Relational MDPs
• Privileged Information Dropout in Reinforcement Learning
• Meta Inverse Reinforcement Learning via Maximum Reward Sharing for Human Motion Analysis
• Near-optimal Representation Learning for Linear Bandits and Linear RL
• Domain Adversarial Reinforcement Learning
• Mutual Information Based Knowledge Transfer Under State-Action Dimension Mismatch
• On the Power of Multitask Representation Learning in Linear MDP
• Knowledge Distillation for Multi-task Learning
• Learning Cross-Domain Correspondence for Control with Dynamics Cycle-Consistency
• AutoSeM: Automatic Task Selection and Mixing in Multi-Task Learning
• Impact of Representation Learning in Linear Bandits
• Review, Analysis and Design of a Comprehensive Deep Reinforcement Learning Framework
• Zero-Shot Learning of Text Adventure Games with Sentence-Level Semantics
• Transferable Cost-Aware Security Policy Implementation for Malware Detection Using Deep Reinforcement Learning
• Multi-task Learning with Gradient Guided Policy Specialization
• SLAW: Scaled Loss Approximate Weighting for Efficient Multi-Task Learning
• Conservative Data Sharing for Multi-Task Offline Reinforcement Learning
• Privacy-Preserving Kickstarting Deep Reinforcement Learning with Privacy-Aware Learners
• Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search
• REPAINT: Knowledge Transfer in Deep Reinforcement Learning
• Learning State Abstractions for Transfer in Continuous Control
• Hierarchically Decoupled Imitation for Morphological Transfer
• Real-time Policy Distillation in Deep Reinforcement Learning
• Exploration for Multi-task Reinforcement Learning with Deep Generative Models
• Fractional Transfer Learning for Deep Model-Based Reinforcement Learning
• Embracing the Dark Knowledge: Domain Generalization Using Regularized Knowledge Distillation
• Self-Organizing Maps as a Storage and Transfer Mechanism in Reinforcement Learning
• Fully Distributed Actor-Critic Architecture for Multitask Deep Reinforcement Learning
• Sparse Attention Guided Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning
• Dynamic Value Estimation for Single-Task Multi-Scene Reinforcement Learning
• Compression and Localization in Reinforcement Learning for ATARI Games
• Faster Reinforcement Learning Using Active Simulators
• Improving image generative models with human interactions
• Composed Fine-Tuning: Freezing Pre-Trained Denoising Autoencoders for Improved Generalization
• Learning Meta Representations for Agents in Multi-Agent Reinforcement Learning
• Scaling shared model governance via model splitting
• Knowledge Distillation-aided End-to-End Learning for Linear Precoding in Multiuser MIMO Downlink Systems with Finite-Rate Feedback
• Prioritized Guidance for Efficient Multi-Agent Reinforcement Learning Exploration
• Don't Forget Your Teacher: A Corrective Reinforcement Learning Framework
• Transfer Learning Across Simulated Robots With Different Sensors
• Decoupled Learning of Environment Characteristics for Safe Exploration
• Is the Meta-Learning Idea Able to Improve the Generalization of Deep Neural Networks on the Standard Supervised Learning?
• When Autonomous Systems Meet Accuracy and Transferability through AI: A Survey
• Learning Shared Dynamics with Meta-World Models
• Transferring Deep Reinforcement Learning with Adversarial Objective and Augmentation
• GalilAI: Out-of-Task Distribution Detection using Causal Active Experimentation for Safe Transfer RL
• Learning Time-Sensitive Strategies in Space Fortress
• Canoe : A System for Collaborative Learning for Neural Nets
• Sequential Dynamic Decision Making with Deep Neural Nets on a Test-Time Budget
• On The Transferability of Deep-Q Networks