Faster CNNs with Direct Sparse Convolutions and Guided Pruning
arXiv:1608.01409
Abstract
Phenomenally successful in practical inference problems, convolutional neural networks (CNN) are widely deployed in mobile devices, data centers, and even supercomputers. The number of parameters needed in CNNs, however, are often large and undesirable. Consequently, various methods have been developed to prune a CNN once it is trained. Nevertheless, the resulting CNNs offer limited benefits. While pruning the fully connected layers reduces a CNN's size considerably, it does not improve inference speed noticeably as the compute heavy parts lie in convolutions. Pruning CNNs in a way that increase inference speed often imposes specific sparsity structures, thus limiting the achievable sparsity levels. We present a method to realize simultaneously size economy and speed improvement while pruning CNNs. Paramount to our success is an efficient general sparse-with-dense matrix multiplication implementation that is applicable to convolution of feature maps with kernels of arbitrary sparsity patterns. Complementing this, we developed a performance model that predicts sweet spots of sparsity levels for different layers and on different computer architectures. Together, these two allow us to demonstrate 3.1--7.3 convolution speedups over dense convolution in AlexNet, on Intel Atom, Xeon, and Xeon Phi processors, spanning the spectrum from mobile devices to supercomputers. We also open source our project at https://github.com/IntelLabs/SkimCaffe.
arXiv comment12 pages, 5 figures
References in corpus (10)
- Caffe: Convolutional Architecture for Fast Feature Embedding
- Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
- End to End Learning for Self-Driving Cars
- Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
- Speeding up Convolutional Neural Networks with Low Rank Expansions
- Learning Structured Sparsity in Deep Neural Networks
- Convolutional neural networks with low-rank regularization
- Fast Algorithms for Convolutional Neural Networks
- Caffe con Troll: Shallow Ideas to Speed Up Deep Learning
- Fast ConvNets Using Group-wise Brain Damage
Cited by in corpus (38)
- SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning
- Enable Deep Learning on Mobile Devices: Methods, Systems, and Applications
- Hardware Acceleration of Sparse and Irregular Tensor Computations of ML Models: A Survey and Insights
- Accelerator-Aware Pruning for Convolutional Neural Networks
- Enabling Sparse Winograd Convolution by Native Pruning
- Distilling Object Detectors with Task Adaptive Regularization
- Low-Memory Neural Network Training: A Technical Report
- Automatic Generation of Efficient Sparse Tensor Format Conversion Routines
- Escoin: Efficient Sparse Convolutional Neural Network Inference on GPUs
- SparseTrain:Leveraging Dynamic Sparsity in Training DNNs on General-Purpose SIMD Processors
- DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures
- HRank: Filter Pruning using High-Rank Feature Map
- Coordinating Filters for Faster Deep Neural Networks
- TIRAMISU: A Polyhedral Compiler for Dense and Sparse Deep Learning
- Intelligent-Unrolling: Exploiting Regular Patterns in Irregular Applications
- Inference, Learning and Attention Mechanisms that Exploit and Preserve Sparsity in Convolutional Networks
- SparseRT: Accelerating Unstructured Sparsity on GPUs for Deep Learning Inference
- Asymptotic Soft Filter Pruning for Deep Convolutional Neural Networks
- RNAS-CL: Robust Neural Architecture Search by Cross-Layer Knowledge Distillation
- Adaptive Dense-to-Sparse Paradigm for Pruning Online Recommendation System with Non-Stationary Data
- AutoGrow: Automatic Layer Growing in Deep Convolutional Networks
- NUQSGD: Provably Communication-efficient Data-parallel SGD via Nonuniform Quantization
- Compression of Deep Convolutional Neural Networks under Joint Sparsity Constraints
- High Performance Convolution Using Sparsity and Patterns for Inference in Deep Convolutional Neural Networks
- SparseDNN: Fast Sparse Deep Learning Inference on CPUs
- AdaptCL: Efficient Collaborative Learning with Dynamic and Adaptive Pruning
- cuConv: A CUDA Implementation of Convolution for CNN Inference
- Accelerated CNN Training Through Gradient Approximation
- Neural Networks at a Fraction with Pruned Quaternions
- Adjoined Networks: A Training Paradigm with Applications to Network Compression
- Channel-wise pruning of neural networks with tapering resource constraint
- Dynamic Sparse Tensor Algebra Compilation
- Deep Hierarchical Machine: a Flexible Divide-and-Conquer Architecture
- Joint Regularization on Activations and Weights for Efficient Neural Network Pruning
- When deep learning models on GPU can be accelerated by taking advantage of unstructured sparsity
- Accelerating convolutional neural network by exploiting sparsity on GPUs
- Hands-on Guidance for Distilling Object Detectors
- LEAN: graph-based pruning for convolutional neural networks by extracting longest chains