The Power of Sparsity in Convolutional Neural Networks

Abstract

Deep convolutional networks are well-known for their high computational and memory demands. Given limited resources, how does one design a network that balances its size, training time, and prediction accuracy? A surprisingly effective approach to trade accuracy for size and speed is to simply reduce the number of channels in each convolutional layer by a fixed fraction and retrain the network. In many cases this leads to significantly smaller networks with only minimal changes to accuracy. In this paper, we take a step further by empirically examining a strategy for deactivating connections between filters in convolutional layers in a way that allows us to harvest savings both in run-time and memory for many network architectures. More specifically, we generalize 2D convolution to use a channel-wise sparse connection structure and show that this leads to significantly better results than the baseline approach for large networks including VGG and Inception V3.

References in corpus (8)

• Improving neural networks by preventing co-adaptation of feature detectors
• Compressing Deep Convolutional Networks using Vector Quantization
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Learning Structured Sparsity in Deep Neural Networks
• High-Performance Neural Networks for Visual Object Classification
• DSD: Dense-Sparse-Dense Training for Deep Neural Networks
• Training Skinny Deep Neural Networks with Iterative Hard Thresholding Methods

Cited by in corpus (14)

• Learning Efficient Convolutional Networks through Network Slimming
• A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference
• Interleaved Group Convolutions for Deep Neural Networks
• Speeding up Convolutional Neural Networks By Exploiting the Sparsity of Rectifier Units
• Optimally Scheduling CNN Convolutions for Efficient Memory Access
• LeanConvNets: Low-cost Yet Effective Convolutional Neural Networks
• High Performance Convolution Using Sparsity and Patterns for Inference in Deep Convolutional Neural Networks
• LeanResNet: A Low-cost Yet Effective Convolutional Residual Networks
• AdaDeep: A Usage-Driven, Automated Deep Model Compression Framework for Enabling Ubiquitous Intelligent Mobiles
• Scaling shared model governance via model splitting
• Magnitude and Uncertainty Pruning Criterion for Neural Networks
• Cascaded Coarse-to-Fine Deep Kernel Networks for Efficient Satellite Image Change Detection
• The Impact of Activation Sparsity on Overfitting in Convolutional Neural Networks
• Training of deep residual networks with stochastic MG/OPT