Neural Rejuvenation: Improving Deep Network Training by Enhancing Computational Resource Utilization
arXiv:1812.00481
Abstract
In this paper, we study the problem of improving computational resource utilization of neural networks. Deep neural networks are usually over-parameterized for their tasks in order to achieve good performances, thus are likely to have underutilized computational resources. This observation motivates a lot of research topics, e.g. network pruning, architecture search, etc. As models with higher computational costs (e.g. more parameters or more computations) usually have better performances, we study the problem of improving the resource utilization of neural networks so that their potentials can be further realized. To this end, we propose a novel optimization method named Neural Rejuvenation. As its name suggests, our method detects dead neurons and computes resource utilization in real time, rejuvenates dead neurons by resource reallocation and reinitialization, and trains them with new training schemes. By simply replacing standard optimizers with Neural Rejuvenation, we are able to improve the performances of neural networks by a very large margin while using similar training efforts and maintaining their original resource usages.
References in corpus (32)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Distilling the Knowledge in a Neural Network
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Deep Residual Learning for Image Recognition
- Neural Architecture Search with Reinforcement Learning
- Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs
- Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
- Densely Connected Convolutional Networks
- Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
- Going Deeper with Convolutions
- Training Very Deep Networks
- Pruning Convolutional Neural Networks for Resource Efficient Inference
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
- Recurrent Models of Visual Attention
- Multiple Object Recognition with Visual Attention
- Pruning Filters for Efficient ConvNets
- Efficient Neural Architecture Search via Parameter Sharing
- Deep Clustering for Unsupervised Learning of Visual Features
- Learning Structured Sparsity in Deep Neural Networks
- Channel Pruning for Accelerating Very Deep Neural Networks
- Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
- Efficient Architecture Search by Network Transformation
- ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering
- Simple And Efficient Architecture Search for Convolutional Neural Networks
- Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
- Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
- Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Students
- Stacked Cross Attention for Image-Text Matching
- Practical Block-wise Neural Network Architecture Generation
- Few-Shot Image Recognition by Predicting Parameters from Activations
- ScaleNet: Guiding Object Proposal Generation in Supermarkets and Beyond