Full deep neural network training on a pruned weight budget

Abstract

We introduce a DNN training technique that learns only a fraction of the full parameter set without incurring an accuracy penalty. To do this, our algorithm constrains the total number of weights updated during backpropagation to those with the highest total gradients. The remaining weights are not tracked, and their initial value is regenerated at every access to avoid storing them in memory. This can dramatically reduce the number of off-chip memory accesses during both training and inference, a key component of the energy needs of DNN accelerators. By ensuring that the total weight diffusion remains close to that of baseline unpruned SGD, networks pruned using our technique are able to retain state-of-the-art accuracy across network architectures -- including networks previously identified as difficult to compress, such as Densenet and WRN. With ResNet18 on ImageNet, we observe an 11.7$\times$ weight reduction with no accuracy loss, and up to 24.4$\times$ with a small accuracy impact.

References in corpus (28)

• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Densely Connected Convolutional Networks
• Wide Residual Networks
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Deep Learning with Limited Numerical Precision
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Compressing Neural Networks with the Hashing Trick
• Training Deep Nets with Sublinear Memory Cost
• Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
• Variational Dropout and the Local Reparameterization Trick
• Train longer, generalize better: closing the generalization gap in large batch training of neural networks
• Revisiting Small Batch Training for Deep Neural Networks
• Training deep neural networks with low precision multiplications
• XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
• Learning Efficient Convolutional Networks through Network Slimming
• Qualitatively characterizing neural network optimization problems
• WRPN: Wide Reduced-Precision Networks
• Learning Sparse Neural Networks through $L_0$ Regularization
• DSD: Dense-Sparse-Dense Training for Deep Neural Networks
• Compression-aware Training of Deep Networks
• ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression
• Ristretto: Hardware-Oriented Approximation of Convolutional Neural Networks
• Deep Learning with Low Precision by Half-wave Gaussian Quantization
• Cnvlutin2: Ineffectual-Activation-and-Weight-Free Deep Neural Network Computing
• Quantized Convolutional Neural Networks for Mobile Devices
• NoiseOut: A Simple Way to Prune Neural Networks
• SparseNN: An Energy-Efficient Neural Network Accelerator Exploiting Input and Output Sparsity

Cited by in corpus (7)

• Artificial Intelligence of Things: A Survey
• ASAP: Architecture Search, Anneal and Prune
• Explaining Deep Learning Models using Causal Inference
• Sparse Weight Activation Training
• A Generalized Lottery Ticket Hypothesis
• Accelerated CNN Training Through Gradient Approximation
• Stochastic Model Pruning via Weight Dropping Away and Back