Memory Bounded Deep Convolutional Networks

Abstract

In this work, we investigate the use of sparsity-inducing regularizers during training of Convolution Neural Networks (CNNs). These regularizers encourage that fewer connections in the convolution and fully connected layers take non-zero values and in effect result in sparse connectivity between hidden units in the deep network. This in turn reduces the memory and runtime cost involved in deploying the learned CNNs. We show that training with such regularization can still be performed using stochastic gradient descent implying that it can be used easily in existing codebases. Experimental evaluation of our approach on MNIST, CIFAR, and ImageNet datasets shows that our regularizers can result in dramatic reductions in memory requirements. For instance, when applied on AlexNet, our method can reduce the memory consumption by a factor of four with minimal loss in accuracy.

References in corpus (4)

• Improving neural networks by preventing co-adaptation of feature detectors
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Going Deeper with Convolutions
• Speeding up Convolutional Neural Networks with Low Rank Expansions

Cited by in corpus (21)

• The State of Sparsity in Deep Neural Networks
• Training Skinny Deep Neural Networks with Iterative Hard Thresholding Methods
• MEC: Memory-efficient Convolution for Deep Neural Network
• Compact Deep Convolutional Neural Networks With Coarse Pruning
• Performance Guaranteed Network Acceleration via High-Order Residual Quantization
• Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure
• Alternating Direction Method of Multipliers for Sparse Convolutional Neural Networks
• Training Sparse Neural Networks
• GASL: Guided Attention for Sparsity Learning in Deep Neural Networks
• Transformed ${\ell }_1$ Regularization for Learning Sparse Deep Neural Networks
• Pruning at a Glance: Global Neural Pruning for Model Compression
• Manipulating Identical Filter Redundancy for Efficient Pruning on Deep and Complicated CNN
• Finding trainable sparse networks through Neural Tangent Transfer
• How Not to Give a FLOP: Combining Regularization and Pruning for Efficient Inference
• ESPN: Extremely Sparse Pruned Networks
• Building Compact and Robust Deep Neural Networks with Toeplitz Matrices
• Disentangling Neural Architectures and Weights: A Case Study in Supervised Classification
• Differentiable Architecture Pruning for Transfer Learning
• What needles do sparse neural networks find in nonlinear haystacks
• The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation
• $S^2$-LBI: Stochastic Split Linearized Bregman Iterations for Parsimonious Deep Learning