Training Larger Networks for Deep Reinforcement Learning
arXiv:2102.07920
Abstract
The success of deep learning in the computer vision and natural language processing communities can be attributed to training of very deep neural networks with millions or billions of parameters which can then be trained with massive amounts of data. However, similar trend has largely eluded training of deep reinforcement learning (RL) algorithms where larger networks do not lead to performance improvement. Previous work has shown that this is mostly due to instability during training of deep RL agents when using larger networks. In this paper, we make an attempt to understand and address training of larger networks for deep RL. We first show that naively increasing network capacity does not improve performance. Then, we propose a novel method that consists of 1) wider networks with DenseNet connection, 2) decoupling representation learning from training of RL, 3) a distributed training method to mitigate overfitting problems. Using this three-fold technique, we show that we can train very large networks that result in significant performance gains. We present several ablation studies to demonstrate the efficacy of the proposed method and some intuitive understanding of the reasons for performance gain. We show that our proposed method outperforms other baseline algorithms on several challenging locomotion tasks.
Under submission
References in corpus (20)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
- Language Models are Few-Shot Learners
- Data-Efficient Image Recognition with Contrastive Predictive Coding
- Searching for Activation Functions
- Distributed Prioritized Experience Replay
- Recurrent World Models Facilitate Policy Evolution
- Rainbow: Combining Improvements in Deep Reinforcement Learning
- Deep Reinforcement Learning that Matters
- State Representation Learning for Control: An Overview
- Reinforcement Learning with Unsupervised Auxiliary Tasks
- Deep Reinforcement Learning and the Deadly Triad
- Accelerated Methods for Deep Reinforcement Learning
- Revisiting Fundamentals of Experience Replay
- Decoupling Representation Learning from Reinforcement Learning
- Towards Characterizing Divergence in Deep Q-Learning
- Diagnosing Bottlenecks in Deep Q-learning Algorithms
- Scaling Laws for Transfer
- Regularization Matters in Policy Optimization
- D2RL: Deep Dense Architectures in Reinforcement Learning