Dynamic Sparse Graph for Efficient Deep Learning
arXiv:1810.00859
Abstract
We propose to execute deep neural networks (DNNs) with dynamic and sparse graph (DSG) structure for compressive memory and accelerative execution during both training and inference. The great success of DNNs motivates the pursuing of lightweight models for the deployment onto embedded devices. However, most of the previous studies optimize for inference while neglect training or even complicate it. Training is far more intractable, since (i) the neurons dominate the memory cost rather than the weights in inference; (ii) the dynamic activation makes previous sparse acceleration via one-off optimization on fixed weight invalid; (iii) batch normalization (BN) is critical for maintaining accuracy while its activation reorganization damages the sparsity. To address these issues, DSG activates only a small amount of neurons with high selectivity at each iteration via a dimension-reduction search (DRS) and obtains the BN compatibility via a double-mask selection (DMS). Experiments show significant memory saving (1.7-4.5x) and operation reduction (2.3-4.4x) with little accuracy loss on various benchmarks.
arXiv commentICLR 2019
References in corpus (28)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
- Wide Residual Networks
- DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
- Pruning Convolutional Neural Networks for Resource Efficient Inference
- AMC: AutoML for Model Compression and Acceleration on Mobile Devices
- Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
- Pruning Filters for Efficient ConvNets
- TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
- Tensorizing Neural Networks
- Don't Decay the Learning Rate, Increase the Batch Size
- Training and Inference with Integers in Deep Neural Networks
- On the importance of single directions for generalization
- All you need is a good init
- Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
- Compression-aware Training of Deep Networks
- meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting
- Ultimate tensorization: compressing convolutional and FC layers alike
- Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks
- A novel channel pruning method for deep neural network compression
- Tensor-Train Recurrent Neural Networks for Video Classification
- ImageNet Training in Minutes
- Discovering Low-Precision Networks Close to Full-Precision Networks for Efficient Embedded Inference
- Layer-compensated Pruning for Resource-constrained Convolutional Neural Networks
- Sparsely-Connected Neural Networks: Towards Efficient VLSI Implementation of Deep Neural Networks