Learning Structured Sparsity in Deep Neural Networks

Abstract

High demand for computation resources severely hinders deployment of large-scale Deep Neural Networks (DNN) in resource constrained devices. In this work, we propose a Structured Sparsity Learning (SSL) method to regularize the structures (i.e., filters, channels, filter shapes, and layer depth) of DNNs. SSL can: (1) learn a compact structure from a bigger DNN to reduce computation cost; (2) obtain a hardware-friendly structured sparsity of DNN to efficiently accelerate the DNNs evaluation. Experimental results show that SSL achieves on average 5.1x and 3.1x speedups of convolutional layer computation of AlexNet against CPU and GPU, respectively, with off-the-shelf libraries. These speedups are about twice speedups of non-structured sparsity; (3) regularize the DNN structure to improve classification accuracy. The results show that for CIFAR-10, regularization on layer depth can reduce 20 layers of a Deep Residual Network (ResNet) to 18 layers while improve the accuracy from 91.25% to 92.60%, which is still slightly higher than that of original ResNet with 32 layers. For AlexNet, structure regularization by SSL also reduces the error by around ~1%. Open source code is in https://github.com/wenwei202/caffe/tree/scnn

Accepted by NIPS 2016

References in corpus (5)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Going Deeper with Convolutions
• cuDNN: Efficient Primitives for Deep Learning
• Speeding up Convolutional Neural Networks with Low Rank Expansions

Cited by in corpus (26)

• ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
• Channel Pruning for Accelerating Very Deep Neural Networks
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
• Learning Efficient Convolutional Networks through Network Slimming
• Exploring the Regularity of Sparse Structure in Convolutional Neural Networks
• Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
• ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression
• Block-Sparse Recurrent Neural Networks
• The Power of Sparsity in Convolutional Neural Networks
• Interleaved Group Convolutions for Deep Neural Networks
• Model Selection in Bayesian Neural Networks via Horseshoe Priors
• Sparsely-Connected Neural Networks: Towards Efficient VLSI Implementation of Deep Neural Networks
• Speeding up Convolutional Neural Networks By Exploiting the Sparsity of Rectifier Units
• Compressing Word Embeddings via Deep Compositional Code Learning
• Evolutionary Synthesis of Deep Neural Networks via Synaptic Cluster-driven Genetic Encoding
• Fine-Pruning: Joint Fine-Tuning and Compression of a Convolutional Network with Bayesian Optimization
• Alternating Direction Method of Multipliers for Sparse Convolutional Neural Networks
• Towards Evolutional Compression
• Dynamic Optimization of Neural Network Structures Using Probabilistic Modeling
• Improved Bayesian Compression
• Ternary Residual Networks
• SparCE: Sparsity aware General Purpose Core Extensions to Accelerate Deep Neural Networks
• Generalized Deep Image to Image Regression
• Incomplete Dot Products for Dynamic Computation Scaling in Neural Network Inference
• Multi-Mode Inference Engine for Convolutional Neural Networks
• Prune the Convolutional Neural Networks with Sparse Shrink