On The Transferability of Deep-Q Networks

Abstract

Transfer Learning (TL) is an efficient machine learning paradigm that allows overcoming some of the hurdles that characterize the successful training of deep neural networks, ranging from long training times to the needs of large datasets. While exploiting TL is a well established and successful training practice in Supervised Learning (SL), its applicability in Deep Reinforcement Learning (DRL) is rarer. In this paper, we study the level of transferability of three different variants of Deep-Q Networks on popular DRL benchmarks as well as on a set of novel, carefully designed control tasks. Our results show that transferring neural networks in a DRL context can be particularly challenging and is a process which in most cases results in negative transfer. In the attempt of understanding why Deep-Q Networks transfer so poorly, we gain novel insights into the training dynamics that characterizes this family of algorithms.

References in corpus (22)

• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Dueling Network Architectures for Deep Reinforcement Learning
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Recurrent Models of Visual Attention
• Model-Based Reinforcement Learning for Atari
• Deep Reinforcement Learning that Matters
• Learning Latent Dynamics for Planning from Pixels
• What makes ImageNet good for transfer learning?
• Sample Efficient Actor-Critic with Experience Replay
• Actor-Mimic: Deep Multitask and Transfer Reinforcement Learning
• Dream to Control: Learning Behaviors by Latent Imagination
• Generalization and Regularization in DQN
• Transfer learning for galaxy morphology from one survey to another
• Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Research
• Improving Computational Efficiency in Visual Reinforcement Learning via Stored Embeddings
• Deep Quality-Value (DQV) Learning
• A Model-based Approach for Sample-efficient Multi-task Reinforcement Learning
• How Transferable are the Representations Learned by Deep Q Agents?
• Fractional Transfer Learning for Deep Model-Based Reinforcement Learning