Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch

Abstract

Sparsity in Deep Neural Networks (DNNs) has been widely studied to compress and accelerate the models on resource-constrained environments. It can be generally categorized into unstructured fine-grained sparsity that zeroes out multiple individual weights distributed across the neural network, and structured coarse-grained sparsity which prunes blocks of sub-networks of a neural network. Fine-grained sparsity can achieve a high compression ratio but is not hardware friendly and hence receives limited speed gains. On the other hand, coarse-grained sparsity cannot concurrently achieve both apparent acceleration on modern GPUs and decent performance. In this paper, we are the first to study training from scratch an N:M fine-grained structured sparse network, which can maintain the advantages of both unstructured fine-grained sparsity and structured coarse-grained sparsity simultaneously on specifically designed GPUs. Specifically, a 2:4 sparse network could achieve 2x speed-up without performance drop on Nvidia A100 GPUs. Furthermore, we propose a novel and effective ingredient, sparse-refined straight-through estimator (SR-STE), to alleviate the negative influence of the approximated gradients computed by vanilla STE during optimization. We also define a metric, Sparse Architecture Divergence (SAD), to measure the sparse network's topology change during the training process. Finally, We justify SR-STE's advantages with SAD and demonstrate the effectiveness of SR-STE by performing comprehensive experiments on various tasks. Source codes and models are available at https://github.com/NM-sparsity/NM-sparsity.

arXiv commentICLR2021

References in corpus (17)

• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Language Models are Few-Shot Learners
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• MMDetection: Open MMLab Detection Toolbox and Benchmark
• Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
• The State of Sparsity in Deep Neural Networks
• Sparse Networks from Scratch: Faster Training without Losing Performance
• Comparing Rewinding and Fine-tuning in Neural Network Pruning
• Deep Rewiring: Training very sparse deep networks
• Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization
• Soft Threshold Weight Reparameterization for Learnable Sparsity
• Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets
• Cognitive Model Priors for Predicting Human Decisions
• The Difficulty of Training Sparse Neural Networks
• PCNN: Pattern-based Fine-Grained Regular Pruning towards Optimizing CNN Accelerators

Cited by in corpus (10)

• An Algorithm-Hardware Co-Optimized Framework for Accelerating N:M Sparse Transformers
• Dissecting Tensor Cores via Microbenchmarks: Latency, Throughput and Numeric Behaviors
• Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks
• Accelerating Sparse Deep Neural Networks
• Sparse Training via Boosting Pruning Plasticity with Neuroregeneration
• Sparse-DySta: Sparsity-Aware Dynamic and Static Scheduling for Sparse Multi-DNN Workloads
• NxMTransformer: Semi-Structured Sparsification for Natural Language Understanding via ADMM
• Towards Structured Dynamic Sparse Pre-Training of BERT
• Search Spaces for Neural Model Training
• Optimizing Connectivity through Network Gradients for Restricted Boltzmann Machines