Learning Structured Sparsity in Deep Neural Networks

Abstract

High demand for computation resources severely hinders deployment of large-scale Deep Neural Networks (DNN) in resource constrained devices. In this work, we propose a Structured Sparsity Learning (SSL) method to regularize the structures (i.e., filters, channels, filter shapes, and layer depth) of DNNs. SSL can: (1) learn a compact structure from a bigger DNN to reduce computation cost; (2) obtain a hardware-friendly structured sparsity of DNN to efficiently accelerate the DNNs evaluation. Experimental results show that SSL achieves on average 5.1x and 3.1x speedups of convolutional layer computation of AlexNet against CPU and GPU, respectively, with off-the-shelf libraries. These speedups are about twice speedups of non-structured sparsity; (3) regularize the DNN structure to improve classification accuracy. The results show that for CIFAR-10, regularization on layer depth can reduce 20 layers of a Deep Residual Network (ResNet) to 18 layers while improve the accuracy from 91.25% to 92.60%, which is still slightly higher than that of original ResNet with 32 layers. For AlexNet, structure regularization by SSL also reduces the error by around ~1%. Open source code is in https://github.com/wenwei202/caffe/tree/scnn

Accepted by NIPS 2016

References in corpus (5)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Going Deeper with Convolutions
• cuDNN: Efficient Primitives for Deep Learning
• Speeding up Convolutional Neural Networks with Low Rank Expansions

Cited by in corpus (40)

• ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
• Channel Pruning for Accelerating Very Deep Neural Networks
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
• Learning Efficient Convolutional Networks through Network Slimming
• Slimmable Neural Networks
• Exploring the Regularity of Sparse Structure in Convolutional Neural Networks
• Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
• ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression
• Block-Sparse Recurrent Neural Networks
• The Power of Sparsity in Convolutional Neural Networks
• Interleaved Group Convolutions for Deep Neural Networks
• Deep Neural Network Approximation for Custom Hardware: Where We've Been, Where We're Going
• Neural Network Distiller: A Python Package For DNN Compression Research
• Model Selection in Bayesian Neural Networks via Horseshoe Priors
• Sparsely-Connected Neural Networks: Towards Efficient VLSI Implementation of Deep Neural Networks
• Speeding up Convolutional Neural Networks By Exploiting the Sparsity of Rectifier Units
• Compressing Word Embeddings via Deep Compositional Code Learning
• Evolutionary Synthesis of Deep Neural Networks via Synaptic Cluster-driven Genetic Encoding
• ChamNet: Towards Efficient Network Design through Platform-Aware Model Adaptation
• Fine-Pruning: Joint Fine-Tuning and Compression of a Convolutional Network with Bayesian Optimization
• Alternating Direction Method of Multipliers for Sparse Convolutional Neural Networks
• Dynamic Optimization of Neural Network Structures Using Probabilistic Modeling
• Towards Evolutional Compression
• Improved Bayesian Compression
• CGaP: Continuous Growth and Pruning for Efficient Deep Learning
• Fractional Skipping: Towards Finer-Grained Dynamic CNN Inference
• Pruning at a Glance: Global Neural Pruning for Model Compression
• Ternary Residual Networks
• SparCE: Sparsity aware General Purpose Core Extensions to Accelerate Deep Neural Networks
• LPRNet: Lightweight Deep Network by Low-rank Pointwise Residual Convolution
• Generalized Deep Image to Image Regression
• Incomplete Dot Products for Dynamic Computation Scaling in Neural Network Inference
• Studying the Plasticity in Deep Convolutional Neural Networks using Random Pruning
• A Layer Decomposition-Recomposition Framework for Neuron Pruning towards Accurate Lightweight Networks
• Multi-Mode Inference Engine for Convolutional Neural Networks
• Multi-Precision Quantized Neural Networks via Encoding Decomposition of -1 and +1
• Cascaded Coarse-to-Fine Deep Kernel Networks for Efficient Satellite Image Change Detection
• Radius Adaptive Convolutional Neural Network
• Adam Induces Implicit Weight Sparsity in Rectifier Neural Networks
• Prune the Convolutional Neural Networks with Sparse Shrink