Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks

Abstract

The growing energy and performance costs of deep learning have driven the community to reduce the size of neural networks by selectively pruning components. Similarly to their biological counterparts, sparse networks generalize just as well, if not better than, the original dense networks. Sparsity can reduce the memory footprint of regular networks to fit mobile devices, as well as shorten training time for ever growing networks. In this paper, we survey prior work on sparsity in deep learning and provide an extensive tutorial of sparsification for both inference and training. We describe approaches to remove and add elements of neural networks, different training strategies to achieve model sparsity, and mechanisms to exploit sparsity in practice. Our work distills ideas from more than 300 research papers and provides guidance to practitioners who wish to utilize sparsity today, as well as to researchers whose goal is to push the frontier forward. We include the necessary background on mathematical methods in sparsification, describe phenomena such as early structure adaptation, the intricate relations between sparsity and the training process, and show techniques for achieving acceleration on real hardware. We also define a metric of pruned parameter efficiency that could serve as a baseline for comparison of different sparse networks. We close by speculating on how sparsity can improve future workloads and outline major open problems in the field.

90 pages, 26 figures

Cited by in corpus (30)

• Structured Pruning for Deep Convolutional Neural Networks: A survey
• Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines
• Unlocking the Emotional World of Visual Media: An Overview of the Science, Research, and Impact of Understanding Emotion
• An Algorithm-Hardware Co-Optimized Framework for Accelerating N:M Sparse Transformers
• Near-Optimal Sparse Allreduce for Distributed Deep Learning
• Flare: Flexible In-Network Allreduce
• TinyVers: A Tiny Versatile System-on-chip with State-Retentive eMRAM for ML Inference at the Extreme Edge
• Resource-Efficient Deep Learning: A Survey on Model-, Arithmetic-, and Implementation-Level Techniques
• Sustainable Supercomputing for AI: GPU Power Capping at HPC Scale
• HighLight: Efficient and Flexible DNN Acceleration with Hierarchical Structured Sparsity
• Efficient Quantized Sparse Matrix Operations on Tensor Cores
• Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning Experiences
• Autoencoding with a Classifier System
• Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modeling
• Sparse deep neural networks for modeling aluminum electrolysis dynamics
• Multiobjective Evolutionary Pruning of Deep Neural Networks with Transfer Learning for improving their Performance and Robustness
• Sparse Stream Semantic Registers: A Lightweight ISA Extension Accelerating General Sparse Linear Algebra
• Acc-SpMM: Accelerating General-purpose Sparse Matrix-Matrix Multiplication with GPU Tensor Cores
• Language Modeling on a SpiNNaker 2 Neuromorphic Chip
• FPGA Resource-aware Structured Pruning for Real-Time Neural Networks
• Talaria: Interactively Optimizing Machine Learning Models for Efficient Inference
• KinePose: A temporally optimized inverse kinematics technique for 6DOF human pose estimation with biomechanical constraints
• Moderator: Moderating Text-to-Image Diffusion Models through Fine-grained Context-based Policies
• Self-building Neural Networks
• Catapult Dynamics and Phase Transitions in Quadratic Nets
• Searching Search Spaces: Meta-evolving a Geometric Encoding for Neural Networks
• Saddle Hierarchy in Dense Associative Memory
• Polynomially Over-Parameterized Convolutional Neural Networks Contain Structured Strong Winning Lottery Tickets
• Induced Feature Selection by Structured Pruning
• Polychrony as Chinampas