DSD: Dense-Sparse-Dense Training for Deep Neural Networks

Abstract

Modern deep neural networks have a large number of parameters, making them very hard to train. We propose DSD, a dense-sparse-dense training flow, for regularizing deep neural networks and achieving better optimization performance. In the first D (Dense) step, we train a dense network to learn connection weights and importance. In the S (Sparse) step, we regularize the network by pruning the unimportant connections with small weights and retraining the network given the sparsity constraint. In the final D (re-Dense) step, we increase the model capacity by removing the sparsity constraint, re-initialize the pruned parameters from zero and retrain the whole dense network. Experiments show that DSD training can improve the performance for a wide range of CNNs, RNNs and LSTMs on the tasks of image classification, caption generation and speech recognition. On ImageNet, DSD improved the Top1 accuracy of GoogLeNet by 1.1%, VGG-16 by 4.3%, ResNet-18 by 1.2% and ResNet-50 by 1.1%, respectively. On the WSJ'93 dataset, DSD improved DeepSpeech and DeepSpeech2 WER by 2.0% and 1.1%. On the Flickr-8K dataset, DSD improved the NeuralTalk BLEU score by over 1.7. DSD is easy to use in practice: at training time, DSD incurs only one extra hyper-parameter: the sparsity ratio in the S step. At testing time, DSD doesn't change the network architecture or incur any inference overhead. The consistent and significant performance gain of DSD experiments shows the inadequacy of the current training methods for finding the best local optimum, while DSD effectively achieves superior optimization performance for finding a better solution. DSD models are available to download at https://songhan.github.io/DSD.

Published as a conference paper at ICLR 2017

References in corpus (10)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Sparsity and Incoherence in Compressive Sampling
• Truncated Power Method for Sparse Eigenvalue Problems
• All you need is a good init
• Training Skinny Deep Neural Networks with Iterative Hard Thresholding Methods
• High Dimensional Expectation-Maximization Algorithm: Statistical Optimization and Asymptotic Normality
• Nonconvex Statistical Optimization: Minimax-Optimal Sparse PCA in Polynomial Time
• Sparse Generalized Eigenvalue Problem: Optimal Statistical Rates via Truncated Rayleigh Flow

Cited by in corpus (47)

• Local Learning with Deep and Handcrafted Features for Facial Expression Recognition
• Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks with Octave Convolution
• ICNet for Real-Time Semantic Segmentation on High-Resolution Images
• The Power of Sparsity in Convolutional Neural Networks
• Revisiting the Importance of Individual Units in CNNs via Ablation
• Optimizing Speech Recognition For The Edge
• Pruning and Quantization for Deep Neural Network Acceleration: A Survey
• Building Efficient ConvNets using Redundant Feature Pruning
• M2KD: Multi-model and Multi-level Knowledge Distillation for Incremental Learning
• A Comprehensive Review and a Taxonomy of Edge Machine Learning: Requirements, Paradigms, and Techniques
• Dynamic Hard Pruning of Neural Networks at the Edge of the Internet
• Grow and Prune Compact, Fast, and Accurate LSTMs
• LeanConvNets: Low-cost Yet Effective Convolutional Neural Networks
• Exploring the Design Space of Deep Convolutional Neural Networks at Large Scale
• Retraining-Based Iterative Weight Quantization for Deep Neural Networks
• AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networks
• Equivalent and Approximate Transformations of Deep Neural Networks
• Serving Recurrent Neural Networks Efficiently with a Spatial Accelerator
• Full deep neural network training on a pruned weight budget
• Taxonomy of Saliency Metrics for Channel Pruning
• Nonparametric Neural Networks
• Hu-Fu: Hardware and Software Collaborative Attack Framework against Neural Networks
• Structurally Sparsified Backward Propagation for Faster Long Short-Term Memory Training
• Composition of Saliency Metrics for Channel Pruning with a Myopic Oracle
• Repulsive Attention: Rethinking Multi-head Attention as Bayesian Inference
• A Simple Method to Reduce Off-chip Memory Accesses on Convolutional Neural Networks
• Pruning Randomly Initialized Neural Networks with Iterative Randomization
• Convolutional Neural Network Quantization using Generalized Gamma Distribution
• The Impact of Reinitialization on Generalization in Convolutional Neural Networks
• Effective Multi-Stage Training Model For Edge Computing Devices In Intrusion Detection
• RePr: Improved Training of Convolutional Filters
• Reviewing continual learning from the perspective of human-level intelligence
• ACDC: Weight Sharing in Atom-Coefficient Decomposed Convolution
• LeanResNet: A Low-cost Yet Effective Convolutional Residual Networks
• Gradient-Coherent Strong Regularization for Deep Neural Networks
• MPDCompress - Matrix Permutation Decomposition Algorithm for Deep Neural Network Compression
• Constrained Differentially Private Federated Learning for Low-bandwidth Devices
• Stochastic Model Pruning via Weight Dropping Away and Back
• Introducing Pose Consistency and Warp-Alignment for Self-Supervised 6D Object Pose Estimation in Color Images
• Multigrid-in-Channels Architectures for Wide Convolutional Neural Networks
• Self-Reorganizing and Rejuvenating CNNs for Increasing Model Capacity Utilization
• Epoch-evolving Gaussian Process Guided Learning
• Light Multi-segment Activation for Model Compression
• Distilling Knowledge From a Deep Pose Regressor Network
• ViP: Virtual Pooling for Accelerating CNN-based Image Classification and Object Detection
• Recent Advances in Large Margin Learning
• Dense-Sparse Deep Convolutional Neural Networks Training for Image Denoising