The State of Sparsity in Deep Neural Networks

Abstract

We rigorously evaluate three state-of-the-art techniques for inducing sparsity in deep neural networks on two large-scale learning tasks: Transformer trained on WMT 2014 English-to-German, and ResNet-50 trained on ImageNet. Across thousands of experiments, we demonstrate that complex techniques (Molchanov et al., 2017; Louizos et al., 2017b) shown to yield high compression rates on smaller datasets perform inconsistently, and that simple magnitude pruning approaches achieve comparable or better results. Additionally, we replicate the experiments performed by (Frankle & Carbin, 2018) and (Liu et al., 2018) at scale and show that unstructured sparse architectures learned through pruning cannot be trained from scratch to the same test set performance as a model trained with joint sparsification and optimization. Together, these results highlight the need for large-scale benchmarks in the field of model compression. We open-source our code, top performing model checkpoints, and results of all hyperparameter configurations to establish rigorous baselines for future work on compression and sparsification.

References in corpus (9)

• Auto-Encoding Variational Bayes
• WaveNet: A Generative Model for Raw Audio
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Bayesian Compression for Deep Learning
• Variational Dropout Sparsifies Deep Neural Networks
• Exploring Sparsity in Recurrent Neural Networks
• Memory Bounded Deep Convolutional Networks
• Faster gaze prediction with dense networks and Fisher pruning
• Compressing Neural Networks using the Variational Information Bottleneck

Cited by in corpus (157)

• Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better
• Integration of Neural Network-Based Symbolic Regression in Deep Learning for Scientific Discovery
• Sparse Networks from Scratch: Faster Training without Losing Performance
• Adaptive Extreme Edge Computing for Wearable Devices
• Stabilizing the Lottery Ticket Hypothesis
• Rigging the Lottery: Making All Tickets Winners
• Hardware Acceleration of Sparse and Irregular Tensor Computations of ML Models: A Survey and Insights
• One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers
• Soft Threshold Weight Reparameterization for Learnable Sparsity
• Linear Mode Connectivity and the Lottery Ticket Hypothesis
• Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP
• Continual Learning via Neural Pruning
• What Do Compressed Deep Neural Networks Forget?
• Dynamic Model Pruning with Feedback
• Chasing Sparsity in Vision Transformers: An End-to-End Exploration
• SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers
• Ansor: Generating High-Performance Tensor Programs for Deep Learning
• Applications and Techniques for Fast Machine Learning in Science
• Pruning neural networks without any data by iteratively conserving synaptic flow
• Learning Sparse Networks Using Targeted Dropout
• Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch
• Structured Pruning of a BERT-based Question Answering Model
• Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned
• Layer-adaptive sparsity for the Magnitude-based Pruning
• Pruning and Quantization for Deep Neural Network Acceleration: A Survey
• Winning the Lottery with Continuous Sparsification
• Pruning Neural Networks at Initialization: Why are We Missing the Mark?
• Characterising Bias in Compressed Models
• Only Train Once: A One-Shot Neural Network Training And Pruning Framework
• Discovering interpretable elastoplasticity models via the neural polynomial method enabled symbolic regressions
• WoodFisher: Efficient Second-Order Approximation for Neural Network Compression
• Complexity-Driven CNN Compression for Resource-constrained Edge AI
• The Lottery Ticket Hypothesis for Pre-trained BERT Networks
• Low-Memory Neural Network Training: A Technical Report
• Movement Pruning: Adaptive Sparsity by Fine-Tuning
• The Lottery Tickets Hypothesis for Supervised and Self-supervised Pre-training in Computer Vision Models
• Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracy
• Compressing RNNs for IoT devices by 15-38x using Kronecker Products
• Rethinking Weight Decay For Efficient Neural Network Pruning
• RNNPool: Efficient Non-linear Pooling for RAM Constrained Inference
• A Unified Lottery Ticket Hypothesis for Graph Neural Networks
• Scaling Laws for Deep Learning
• Tight Compression: Compressing CNN Through Fine-Grained Pruning and Weight Permutation for Efficient Implementation
• Looking GLAMORous: Vehicle Re-Id in Heterogeneous Cameras Networks with Global and Local Attention
• Provable Filter Pruning for Efficient Neural Networks
• A Closer Look at Structured Pruning for Neural Network Compression
• A Brain-inspired Algorithm for Training Highly Sparse Neural Networks
• SeReNe: Sensitivity based Regularization of Neurons for Structured Sparsity in Neural Networks
• Sparse Transfer Learning via Winning Lottery Tickets
• Data-Independent Neural Pruning via Coresets
• Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training
• $L_0$-ARM: Network Sparsification via Stochastic Binary Optimization
• On the Predictability of Pruning Across Scales
• Lookahead: A Far-Sighted Alternative of Magnitude-based Pruning
• Network Pruning That Matters: A Case Study on Retraining Variants
• Selfish Sparse RNN Training
• Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimization
• SiPPing Neural Networks: Sensitivity-informed Provable Pruning of Neural Networks
• Taxonomy and Evaluation of Structured Compression of Convolutional Neural Networks
• One-Shot Pruning of Recurrent Neural Networks by Jacobian Spectrum Evaluation
• Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot
• Pruning via Iterative Ranking of Sensitivity Statistics
• Sparse Training via Boosting Pruning Plasticity with Neuroregeneration
• PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition
• Directional Pruning of Deep Neural Networks
• Deep Ensembling with No Overhead for either Training or Testing: The All-Round Blessings of Dynamic Sparsity
• AC/DC: Alternating Compressed/DeCompressed Training of Deep Neural Networks
• TIRAMISU: A Polyhedral Compiler for Dense and Sparse Deep Learning
• Improving Neural Network Training using Dynamic Learning Rate Schedule for PINNs and Image Classification
• CAT: Compression-Aware Training for bandwidth reduction
• Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot?
• Adapting by Pruning: A Case Study on BERT
• SQuantizer: Simultaneous Learning for Both Sparse and Low-precision Neural Networks
• Robustness to Pruning Predicts Generalization in Deep Neural Networks
• Neural 3D Scene Compression via Model Compression
• Campfire: Compressible, Regularization-Free, Structured Sparse Training for Hardware Accelerators
• Privacy-preserving Learning via Deep Net Pruning
• Lightweight Convolutional Representations for On-Device Natural Language Processing
• SparseRT: Accelerating Unstructured Sparsity on GPUs for Deep Learning Inference
• word2ket: Space-efficient Word Embeddings inspired by Quantum Entanglement
• Powerpropagation: A sparsity inducing weight reparameterisation
• Training Deep Neural Networks with Joint Quantization and Pruning of Weights and Activations
• Learning to Prune Deep Neural Networks via Reinforcement Learning
• Quantile regression with deep ReLU Networks: Estimators and minimax rates
• Data-Efficient GAN Training Beyond (Just) Augmentations: A Lottery Ticket Perspective
• Non-Differentiable Supervised Learning with Evolution Strategies and Hybrid Methods
• Revisiting Loss Modelling for Unstructured Pruning
• Rethinking Network Pruning -- under the Pre-train and Fine-tune Paradigm
• Neural Network Compression Via Sparse Optimization
• You are caught stealing my winning lottery ticket! Making a lottery ticket claim its ownership
• Robustness in Compressed Neural Networks for Object Detection
• Towards Understanding Iterative Magnitude Pruning: Why Lottery Tickets Win
• Hessian-Aware Pruning and Optimal Neural Implant
• Landscape Connectivity and Dropout Stability of SGD Solutions for Over-parameterized Neural Networks
• GANs Can Play Lottery Tickets Too
• Discovery of slow variables in a class of multiscale stochastic systems via neural networks
• A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustness
• On Iterative Neural Network Pruning, Reinitialization, and the Similarity of Masks
• Data-Independent Structured Pruning of Neural Networks via Coresets
• Topological Insights into Sparse Neural Networks
• Neural Pruning via Growing Regularization
• Connectivity Matters: Neural Network Pruning Through the Lens of Effective Sparsity
• Dynamical Isometry: The Missing Ingredient for Neural Network Pruning
• Effective Sparsification of Neural Networks with Global Sparsity Constraint
• SEVEN: Pruning Transformer Model by Reserving Sentinels
• HALP: Hardware-Aware Latency Pruning
• Learning Sparse & Ternary Neural Networks with Entropy-Constrained Trained Ternarization (EC2T)
• SparseDNN: Fast Sparse Deep Learning Inference on CPUs
• Understanding the effect of sparsity on neural networks robustness
• When to Prune? A Policy towards Early Structural Pruning
• On the Robustness of Monte Carlo Dropout Trained with Noisy Labels
• Training Deep Spiking Auto-encoders without Bursting or Dying Neurons through Regularization
• PrunedCaps: A Case For Primary Capsules Discrimination
• Can we avoid Double Descent in Deep Neural Networks?
• Are wider nets better given the same number of parameters?
• Small, Accurate, and Fast Vehicle Re-ID on the Edge: the SAFR Approach
• Dissecting Pruned Neural Networks
• Bespoke vs. Prêt-à-Porter Lottery Tickets: Exploiting Mask Similarity for Trainable Sub-Network Finding
• Towards Structured Dynamic Sparse Pre-Training of BERT
• Spending Your Winning Lottery Better After Drawing It
• Training for temporal sparsity in deep neural networks, application in video processing
• A fast asynchronous MCMC sampler for sparse Bayesian inference
• An Expectation-Maximization Perspective on Federated Learning
• The Flip Side of the Reweighted Coin: Duality of Adaptive Dropout and Regularization
• Adaptive Variational Bayesian Inference for Sparse Deep Neural Network
• On the Downstream Performance of Compressed Word Embeddings
• Doping: A technique for efficient compression of LSTM models using sparse structured additive matrices
• Convolutional neural networks compression with low rank and sparse tensor decompositions
• The curious case of developmental BERTology: On sparsity, transfer learning, generalization and the brain
• Adjoined Networks: A Training Paradigm with Applications to Network Compression
• Ultra-light deep MIR by trimming lottery tickets
• Grassmannian Packings in Neural Networks: Learning with Maximal Subspace Packings for Diversity and Anti-Sparsity
• The Elastic Lottery Ticket Hypothesis
• Self-building Neural Networks
• EPIC TTS Models: Empirical Pruning Investigations Characterizing Text-To-Speech Models
• Search Spaces for Neural Model Training
• Green DetNet: Computation and Memory efficient DetNet using Smart Compression and Training
• Token Pooling in Vision Transformers
• Now that I can see, I can improve: Enabling data-driven finetuning of CNNs on the edge
• Sifting out the features by pruning: Are convolutional networks the winning lottery ticket of fully connected ones?
• One-Cycle Pruning: Pruning ConvNets Under a Tight Training Budget
• The Efficiency Misnomer
• How Well Do Sparse Imagenet Models Transfer?
• Toward Compact Parameter Representations for Architecture-Agnostic Neural Network Compression
• Induced Feature Selection by Structured Pruning
• Statistical Mechanical Analysis of Neural Network Pruning
• Bayesian Sparsification Methods for Deep Complex-valued Networks
• Adapting the Function Approximation Architecture in Online Reinforcement Learning
• Dep-$L_0$: Improving $L_0$-based Network Sparsification via Dependency Modeling
• The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation
• Performance optimizations on deep noise suppression models
• Probabilistic fine-tuning of pruning masks and PAC-Bayes self-bounded learning
• Deconstructing the Structure of Sparse Neural Networks
• Training Sparse Neural Network by Constraining Synaptic Weight on Unit Lp Sphere
• The Impact of Activation Sparsity on Overfitting in Convolutional Neural Networks
• Dense for the Price of Sparse: Improved Performance of Sparsely Initialized Networks via a Subspace Offset
• Neural networks adapting to datasets: learning network size and topology