Learning Structured Sparsity in Deep Neural Networks

Abstract

High demand for computation resources severely hinders deployment of large-scale Deep Neural Networks (DNN) in resource constrained devices. In this work, we propose a Structured Sparsity Learning (SSL) method to regularize the structures (i.e., filters, channels, filter shapes, and layer depth) of DNNs. SSL can: (1) learn a compact structure from a bigger DNN to reduce computation cost; (2) obtain a hardware-friendly structured sparsity of DNN to efficiently accelerate the DNNs evaluation. Experimental results show that SSL achieves on average 5.1x and 3.1x speedups of convolutional layer computation of AlexNet against CPU and GPU, respectively, with off-the-shelf libraries. These speedups are about twice speedups of non-structured sparsity; (3) regularize the DNN structure to improve classification accuracy. The results show that for CIFAR-10, regularization on layer depth can reduce 20 layers of a Deep Residual Network (ResNet) to 18 layers while improve the accuracy from 91.25% to 92.60%, which is still slightly higher than that of original ResNet with 32 layers. For AlexNet, structure regularization by SSL also reduces the error by around ~1%. Open source code is in https://github.com/wenwei202/caffe/tree/scnn

Accepted by NIPS 2016

References in corpus (7)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Going Deeper with Convolutions
• cuDNN: Efficient Primitives for Deep Learning
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Convolutional neural networks with low-rank regularization

Cited by in corpus (178)

• Pruning Convolutional Neural Networks for Resource Efficient Inference
• ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers
• Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations
• Channel Pruning for Accelerating Very Deep Neural Networks
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
• Bayesian Compression for Deep Learning
• Variational Dropout Sparsifies Deep Neural Networks
• Learning Efficient Convolutional Networks through Network Slimming
• Model compression via distillation and quantization
• ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
• Slimmable Neural Networks
• Exploring the Regularity of Sparse Structure in Convolutional Neural Networks
• Lifelong Learning with Dynamically Expandable Networks
• Discrimination-aware Channel Pruning for Deep Neural Networks
• Discrimination-aware Network Pruning for Deep Model Compression
• Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
• Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
• Learning Sparse Neural Networks through $L_0$ Regularization
• Faster CryptoNets: Leveraging Sparsity for Real-World Encrypted Inference
• Faster CNNs with Direct Sparse Convolutions and Guided Pruning
• Compression-aware Training of Deep Networks
• Compressing Neural Networks using the Variational Information Bottleneck
• Learning Intrinsic Sparse Structures within Long Short-Term Memory
• Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask
• ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression
• Block-Sparse Recurrent Neural Networks
• The Power of Sparsity in Convolutional Neural Networks
• What Do Compressed Deep Neural Networks Forget?
• Interleaved Group Convolutions for Deep Neural Networks
• Network Decoupling: From Regular to Depthwise Separable Convolutions
• Deep Neural Network Approximation for Custom Hardware: Where We've Been, Where We're Going
• Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks
• NISP: Pruning Networks using Neuron Importance Score Propagation
• Accelerating Large Scale Real-Time GNN Inference using Channel Pruning
• Measuring the Intrinsic Dimension of Objective Landscapes
• Structured Probabilistic Pruning for Convolutional Neural Network Acceleration
• Neural Network Distiller: A Python Package For DNN Compression Research
• Efficient Processing of Deep Neural Networks: A Tutorial and Survey
• Enabling Sparse Winograd Convolution by Native Pruning
• Model Selection in Bayesian Neural Networks via Horseshoe Priors
• 2PFPCE: Two-Phase Filter Pruning Based on Conditional Entropy
• ShuffleSeg: Real-time Semantic Segmentation Network
• DAIS: Automatic Channel Pruning via Differentiable Annealing Indicator Search
• Only Train Once: A One-Shot Neural Network Training And Pruning Framework
• A Unified Framework of DNN Weight Pruning and Weight Clustering/Quantization Using ADMM
• You Only Search Once: Single Shot Neural Architecture Search via Direct Sparse Optimization
• Improving Efficiency in Convolutional Neural Network with Multilinear Filters
• Progressive Weight Pruning of Deep Neural Networks using ADMM
• AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Rates
• Sparsely-Connected Neural Networks: Towards Efficient VLSI Implementation of Deep Neural Networks
• Speeding up Convolutional Neural Networks By Exploiting the Sparsity of Rectifier Units
• Gradual Channel Pruning while Training using Feature Relevance Scores for Convolutional Neural Networks
• Compressing Word Embeddings via Deep Compositional Code Learning
• Data-Driven Sparse Structure Selection for Deep Neural Networks
• IGCV3: Interleaved Low-Rank Group Convolutions for Efficient Deep Neural Networks
• Continual Learning with Node-Importance based Adaptive Group Sparse Regularization
• Structured Variational Learning of Bayesian Neural Networks with Horseshoe Priors
• Towards Effective Low-bitwidth Convolutional Neural Networks
• Packing Sparse Convolutional Neural Networks for Efficient Systolic Array Implementations: Column Combining Under Joint Optimization
• Systematic Weight Pruning of DNNs using Alternating Direction Method of Multipliers
• Model Slicing for Supporting Complex Analytics with Elastic Inference Cost and Resource Constraints
• Theoretical Properties for Neural Networks with Weight Matrices of Low Displacement Rank
• Compressing deep quaternion neural networks with targeted regularization
• Fast On-the-fly Retraining-free Sparsification of Convolutional Neural Networks
• $L_0$-ARM: Network Sparsification via Stochastic Binary Optimization
• Evolutionary Synthesis of Deep Neural Networks via Synaptic Cluster-driven Genetic Encoding
• ChamNet: Towards Efficient Network Design through Platform-Aware Model Adaptation
• ThUnderVolt: Enabling Aggressive Voltage Underscaling and Timing Error Resilience for Energy Efficient Deep Neural Network Accelerators
• Prediction of hierarchical time series using structured regularization and its application to artificial neural networks
• Dynamic Deep Neural Networks: Optimizing Accuracy-Efficiency Trade-offs by Selective Execution
• Fine-Pruning: Joint Fine-Tuning and Compression of a Convolutional Network with Bayesian Optimization
• SBNet: Sparse Blocks Network for Fast Inference
• Accelerating Sparse Deep Neural Networks
• Orchestrating the Development Lifecycle of Machine Learning-Based IoT Applications: A Taxonomy and Survey
• Alternating Direction Method of Multipliers for Sparse Convolutional Neural Networks
• Pruning via Iterative Ranking of Sensitivity Statistics
• Towards Evolutional Compression
• Dynamic Optimization of Neural Network Structures Using Probabilistic Modeling
• LCNN: Lookup-based Convolutional Neural Network
• Improved Bayesian Compression
• Coordinating Filters for Faster Deep Neural Networks
• CGaP: Continuous Growth and Pruning for Efficient Deep Learning
• Real-Time Execution of Large-scale Language Models on Mobile
• Patch Slimming for Efficient Vision Transformers
• Taxonomy of Saliency Metrics for Channel Pruning
• Efficient Neural Architecture Search via Proximal Iterations
• Regularizing by the Variance of the Activations' Sample-Variances
• Learning Versatile Convolution Filters for Efficient Visual Recognition
• Characterising Across-Stack Optimisations for Deep Convolutional Neural Networks
• Structurally Sparsified Backward Propagation for Faster Long Short-Term Memory Training
• Fractional Skipping: Towards Finer-Grained Dynamic CNN Inference
• PCNN: Pattern-based Fine-Grained Regular Pruning towards Optimizing CNN Accelerators
• Pruning at a Glance: Global Neural Pruning for Model Compression
• Enabling Retrain-free Deep Neural Network Pruning using Surrogate Lagrangian Relaxation
• Long Short-Term Attention
• Adaptive Dense-to-Sparse Paradigm for Pruning Online Recommendation System with Non-Stationary Data
• Ternary Residual Networks
• AutoGrow: Automatic Layer Growing in Deep Convolutional Networks
• Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling
• Demystifying Neural Network Filter Pruning
• Learning Sparse Neural Networks via Sensitivity-Driven Regularization
• Combinatorial Designs for Deep Learning
• SparCE: Sparsity aware General Purpose Core Extensions to Accelerate Deep Neural Networks
• Compression of Deep Convolutional Neural Networks under Joint Sparsity Constraints
• Spectral Pruning: Compressing Deep Neural Networks via Spectral Analysis and its Generalization Error
• Improving Accuracy of Binary Neural Networks using Unbalanced Activation Distribution
• BARS: Joint Search of Cell Topology and Layout for Accurate and Efficient Binary ARchitectures
• LPRNet: Lightweight Deep Network by Low-rank Pointwise Residual Convolution
• Differentiable Sparsification for Deep Neural Networks
• Enhancing the Regularization Effect of Weight Pruning in Artificial Neural Networks
• Generalized Deep Image to Image Regression
• DeepLight: Deep Lightweight Feature Interactions for Accelerating CTR Predictions in Ad Serving
• A Winning Hand: Compressing Deep Networks Can Improve Out-Of-Distribution Robustness
• A High-Performance Adaptive Quantization Approach for Edge CNN Applications
• Doubly Nested Network for Resource-Efficient Inference
• Detecting Dead Weights and Units in Neural Networks
• Low-Cost Parameterizations of Deep Convolutional Neural Networks
• A Generalized Lottery Ticket Hypothesis
• Generalized Bayesian Posterior Expectation Distillation for Deep Neural Networks
• Learning Sparse Deep Feedforward Networks via Tree Skeleton Expansion
• Target Aware Network Adaptation for Efficient Representation Learning
• Graph-Adaptive Pruning for Efficient Inference of Convolutional Neural Networks
• Impostor Networks for Fast Fine-Grained Recognition
• NeRV: Neural Representations for Videos
• Block-wise Dynamic Sparseness
• AACP: Model Compression by Accurate and Automatic Channel Pruning
• Recent Advances in Efficient Computation of Deep Convolutional Neural Networks
• Deep Neural Compression Via Concurrent Pruning and Self-Distillation
• A Real-time Low-cost Artificial Intelligence System for Autonomous Spraying in Palm Plantations
• Activation Density driven Energy-Efficient Pruning in Training
• DECORE: Deep Compression with Reinforcement Learning
• Recent Advances in Convolutional Neural Network Acceleration
• Incomplete Dot Products for Dynamic Computation Scaling in Neural Network Inference
• Scalable Neural Network Compression and Pruning Using Hard Clustering and L1 Regularization
• Parameter Transfer Unit for Deep Neural Networks
• DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models
• Sparsity in Deep Neural Networks - An Empirical Investigation with TensorQuant
• MPDCompress - Matrix Permutation Decomposition Algorithm for Deep Neural Network Compression
• Automatic Inference of Cross-modal Connection Topologies for X-CNNs
• Convolutional neural networks compression with low rank and sparse tensor decompositions
• Cross-Channel Intragroup Sparsity Neural Network
• Introducing Pose Consistency and Warp-Alignment for Self-Supervised 6D Object Pose Estimation in Color Images
• Quantization Mimic: Towards Very Tiny CNN for Object Detection
• Efficient and Robust Machine Learning for Real-World Systems
• Neural Rejuvenation: Improving Deep Network Training by Enhancing Computational Resource Utilization
• Multi-Mode Inference Engine for Convolutional Neural Networks
• Training for temporal sparsity in deep neural networks, application in video processing
• Studying the Plasticity in Deep Convolutional Neural Networks using Random Pruning
• Sparse Persistent RNNs: Squeezing Large Recurrent Networks On-Chip
• Composite Binary Decomposition Networks
• Finding Everything within Random Binary Networks
• A Layer Decomposition-Recomposition Framework for Neuron Pruning towards Accurate Lightweight Networks
• Recurrent Residual Module for Fast Inference in Videos
• Prune the Convolutional Neural Networks with Sparse Shrink
• Deep Learning Approximation: Zero-Shot Neural Network Speedup
• Learning sparse transformations through backpropagation
• Architecture-aware Network Pruning for Vision Quality Applications
• Approximate Random Dropout
• Against Membership Inference Attack: Pruning is All You Need
• Sparse Architectures for Text-Independent Speaker Verification Using Deep Neural Networks
• Neural Plasticity Networks
• Attribute Guided Sparse Tensor-Based Model for Person Re-Identification
• Building Sparse Deep Feedforward Networks using Tree Receptive Fields
• Learning Sparse Structured Ensembles with SG-MCMC and Network Pruning
• Structured in Space, Randomized in Time: Leveraging Dropout in RNNs for Efficient Training
• Network Adjustment: Channel Search Guided by FLOPs Utilization Ratio
• Towards Real-Time DNN Inference on Mobile Platforms with Model Pruning and Compiler Optimization
• Light Multi-segment Activation for Model Compression
• Radius Adaptive Convolutional Neural Network
• Semi-Relaxed Quantization with DropBits: Training Low-Bit Neural Networks via Bit-wise Regularization
• Adam Induces Implicit Weight Sparsity in Rectifier Neural Networks
• Cascaded Coarse-to-Fine Deep Kernel Networks for Efficient Satellite Image Change Detection
• Efficient Fusion of Sparse and Complementary Convolutions
• Multi-Precision Quantized Neural Networks via Encoding Decomposition of -1 and +1
• Penetrating the Fog: the Path to Efficient CNN Models