Sparse Weight Activation Training

Abstract

Neural network training is computationally and memory intensive. Sparse training can reduce the burden on emerging hardware platforms designed to accelerate sparse computations, but it can affect network convergence. In this work, we propose a novel CNN training algorithm Sparse Weight Activation Training (SWAT). SWAT is more computation and memory-efficient than conventional training. SWAT modifies back-propagation based on the empirical insight that convergence during training tends to be robust to the elimination of (i) small magnitude weights during the forward pass and (ii) both small magnitude weights and activations during the backward pass. We evaluate SWAT on recent CNN architectures such as ResNet, VGG, DenseNet and WideResNet using CIFAR-10, CIFAR-100 and ImageNet datasets. For ResNet-50 on ImageNet SWAT reduces total floating-point operations (FLOPS) during training by 80% resulting in a 3.3$\times$ training speedup when run on a simulated sparse learning accelerator representative of emerging platforms while incurring only 1.63% reduction in validation accuracy. Moreover, SWAT reduces memory footprint during the backward pass by 23% to 50% for activations and 50% to 90% for weights.

Published at NeurIPS 2020

References in corpus (44)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Wide Residual Networks
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• A guide to convolution arithmetic for deep learning
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• Ternary Weight Networks
• Trained Ternary Quantization
• Pruning Filters for Efficient ConvNets
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Tensorizing Neural Networks
• Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Science
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• Towards Accurate Binary Convolutional Neural Network
• Variational Dropout Sparsifies Deep Neural Networks
• Training and Inference with Integers in Deep Neural Networks
• Exploring the Regularity of Sparse Structure in Convolutional Neural Networks
• Training Deep Neural Networks with 8-bit Floating Point Numbers
• Scalable Methods for 8-bit Training of Neural Networks
• Sparse Networks from Scratch: Faster Training without Losing Performance
• Improving Neural Network Quantization without Retraining using Outlier Channel Splitting
• Compression-aware Training of Deep Networks
• The Expressive Power of Neural Networks: A View from the Width
• Mixed Precision Training of Convolutional Neural Networks using Integer Operations
• Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization
• meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting
• Ultimate tensorization: compressing convolutional and FC layers alike
• Soft Threshold Weight Reparameterization for Learnable Sparsity
• Loss-aware Weight Quantization of Deep Networks
• High-Accuracy Low-Precision Training
• Discovering Low-Precision Networks Close to Full-Precision Networks for Efficient Embedded Inference
• LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks
• Training wide residual networks for deployment using a single bit for each weight
• Bridging the Accuracy Gap for 2-bit Quantized Neural Networks (QNN)
• Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers
• On the Expressive Power of Overlapping Architectures of Deep Learning
• Minimal Effort Back Propagation for Convolutional Neural Networks
• Full deep neural network training on a pruned weight budget
• Accelerating Convolutional Neural Networks via Activation Map Compression

Cited by in corpus (7)

• Chasing Sparsity in Vision Transformers: An End-to-End Exploration
• A Brain-inspired Algorithm for Training Highly Sparse Neural Networks
• Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training
• Selfish Sparse RNN Training
• Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity
• Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models
• SMOF: Squeezing More Out of Filters Yields Hardware-Friendly CNN Pruning