AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networks

Abstract

The increasing computational requirements of deep neural networks (DNNs) have led to significant interest in obtaining DNN models that are sparse, yet accurate. Recent work has investigated the even harder case of sparse training, where the DNN weights are, for as much as possible, already sparse to reduce computational costs during training. Existing sparse training methods are often empirical and can have lower accuracy relative to the dense baseline. In this paper, we present a general approach called Alternating Compressed/DeCompressed (AC/DC) training of DNNs, demonstrate convergence for a variant of the algorithm, and show that AC/DC outperforms existing sparse training methods in accuracy at similar computational budgets; at high sparsity levels, AC/DC even outperforms existing methods that rely on accurate pre-trained dense models. An important property of AC/DC is that it allows co-training of dense and sparse models, yielding accurate sparse-dense model pairs at the end of the training process. This is useful in practice, where compressed variants may be desirable for deployment in resource-constrained settings without re-doing the entire training flow, and also provides us with insights into the accuracy gap between dense and compressed models. The code is available at: https://github.com/IST-DASLab/ACDC .

arXiv commentAccepted at NeurIPS 2021

References in corpus (23)

• Adam: A Method for Stochastic Optimization
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• A Convergence Theory for Deep Learning via Over-Parameterization
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science
• Pointer Sentinel Mixture Models
• The State of Sparsity in Deep Neural Networks
• Variational Dropout Sparsifies Deep Neural Networks
• Sparse Networks from Scratch: Faster Training without Losing Performance
• TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems
• Deep Rewiring: Training very sparse deep networks
• DSD: Dense-Sparse-Dense Training for Deep Neural Networks
• Stabilizing the Lottery Ticket Hypothesis
• Soft Threshold Weight Reparameterization for Learnable Sparsity
• Dynamic Model Pruning with Feedback
• Gradient Hard Thresholding Pursuit for Sparsity-Constrained Optimization
• On Iterative Hard Thresholding Methods for High-dimensional M-Estimation
• Understanding Top-k Sparsification in Distributed Deep Learning
• Training Skinny Deep Neural Networks with Iterative Hard Thresholding Methods
• Top-KAST: Top-K Always Sparse Training
• Accelerating Sparse Deep Neural Networks

Cited by in corpus (3)

• Towards Structured Dynamic Sparse Pre-Training of BERT
• How Well Do Sparse Imagenet Models Transfer?
• Masked Training of Neural Networks with Partial Gradients