The Power of Sparsity in Convolutional Neural Networks

Abstract

Deep convolutional networks are well-known for their high computational and memory demands. Given limited resources, how does one design a network that balances its size, training time, and prediction accuracy? A surprisingly effective approach to trade accuracy for size and speed is to simply reduce the number of channels in each convolutional layer by a fixed fraction and retrain the network. In many cases this leads to significantly smaller networks with only minimal changes to accuracy. In this paper, we take a step further by empirically examining a strategy for deactivating connections between filters in convolutional layers in a way that allows us to harvest savings both in run-time and memory for many network architectures. More specifically, we generalize 2D convolution to use a channel-wise sparse connection structure and show that this leads to significantly better results than the baseline approach for large networks including VGG and Inception V3.

References in corpus (17)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Improving neural networks by preventing co-adaptation of feature detectors
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Densely Connected Convolutional Networks
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• Do Deep Nets Really Need to be Deep?
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Compressing Deep Convolutional Networks using Vector Quantization
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• Tensorizing Neural Networks
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Learning Structured Sparsity in Deep Neural Networks
• Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups
• Deep Networks with Stochastic Depth
• High-Performance Neural Networks for Visual Object Classification
• DSD: Dense-Sparse-Dense Training for Deep Neural Networks
• Training Skinny Deep Neural Networks with Iterative Hard Thresholding Methods

Cited by in corpus (30)

• MobileNetV2: Inverted Residuals and Linear Bottlenecks
• Variational Dropout Sparsifies Deep Neural Networks
• Learning Efficient Convolutional Networks through Network Slimming
• A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference
• Deep $k$-Means: Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutions
• Interleaved Group Convolutions for Deep Neural Networks
• What is the State of Neural Network Pruning?
• Building Efficient ConvNets using Redundant Feature Pruning
• s-LWSR: Super Lightweight Super-Resolution Network
• DCFNet: Deep Neural Network with Decomposed Convolutional Filters
• Speeding up Convolutional Neural Networks By Exploiting the Sparsity of Rectifier Units
• Compressing RNNs for IoT devices by 15-38x using Kronecker Products
• Optimally Scheduling CNN Convolutions for Efficient Memory Access
• Model Slicing for Supporting Complex Analytics with Elastic Inference Cost and Resource Constraints
• ANTNets: Mobile Convolutional Neural Networks for Resource Efficient Image Classification
• LeanConvNets: Low-cost Yet Effective Convolutional Neural Networks
• SpecNet: Spectral Domain Convolutional Neural Network
• High Performance Convolution Using Sparsity and Patterns for Inference in Deep Convolutional Neural Networks
• Combinatorial Designs for Deep Learning
• LeanResNet: A Low-cost Yet Effective Convolutional Residual Networks
• AdaDeep: A Usage-Driven, Automated Deep Model Compression Framework for Enabling Ubiquitous Intelligent Mobiles
• Implementing Push-Pull Efficiently in GraphBLAS
• Improving Network Slimming with Nonconvex Regularization
• Multigrid-in-Channels Architectures for Wide Convolutional Neural Networks
• Scaling shared model governance via model splitting
• Recurrent Residual Module for Fast Inference in Videos
• Cascaded Coarse-to-Fine Deep Kernel Networks for Efficient Satellite Image Change Detection
• The Impact of Activation Sparsity on Overfitting in Convolutional Neural Networks
• Magnitude and Uncertainty Pruning Criterion for Neural Networks
• Training of deep residual networks with stochastic MG/OPT