Memory Bounded Deep Convolutional Networks

Abstract

In this work, we investigate the use of sparsity-inducing regularizers during training of Convolution Neural Networks (CNNs). These regularizers encourage that fewer connections in the convolution and fully connected layers take non-zero values and in effect result in sparse connectivity between hidden units in the deep network. This in turn reduces the memory and runtime cost involved in deploying the learned CNNs. We show that training with such regularization can still be performed using stochastic gradient descent implying that it can be used easily in existing codebases. Experimental evaluation of our approach on MNIST, CIFAR, and ImageNet datasets shows that our regularizers can result in dramatic reductions in memory requirements. For instance, when applied on AlexNet, our method can reduce the memory consumption by a factor of four with minimal loss in accuracy.

References in corpus (5)

• Improving neural networks by preventing co-adaptation of feature detectors
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Going Deeper with Convolutions
• OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks
• Speeding up Convolutional Neural Networks with Low Rank Expansions

Cited by in corpus (54)

• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Learning both Weights and Connections for Efficient Neural Networks
• Beyond Sharing Weights for Deep Domain Adaptation
• The State of Sparsity in Deep Neural Networks
• Compression-aware Training of Deep Networks
• Structured Transforms for Small-Footprint Deep Learning
• What Do Compressed Deep Neural Networks Forget?
• Deep neural networks are robust to weight binarization and other non-linear distortions
• Training Skinny Deep Neural Networks with Iterative Hard Thresholding Methods
• NeST: A Neural Network Synthesis Tool Based on a Grow-and-Prune Paradigm
• MEC: Memory-efficient Convolution for Deep Neural Network
• An exploration of parameter redundancy in deep networks with circulant projections
• Structured Pruning of Deep Convolutional Neural Networks
• Compact Deep Convolutional Neural Networks With Coarse Pruning
• Accelerating Very Deep Convolutional Networks for Classification and Detection
• Adversarial Neural Pruning with Latent Vulnerability Suppression
• ACDC: A Structured Efficient Linear Layer
• Performance Guaranteed Network Acceleration via High-Order Residual Quantization
• Fast ConvNets Using Group-wise Brain Damage
• Deep Fried Convnets
• Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure
• Alternating Direction Method of Multipliers for Sparse Convolutional Neural Networks
• Constrained Deep Learning using Conditional Gradient and Applications in Computer Vision
• Training Sparse Neural Networks
• GASL: Guided Attention for Sparsity Learning in Deep Neural Networks
• Transformed ${\ell }_1$ Regularization for Learning Sparse Deep Neural Networks
• Pruning at a Glance: Global Neural Pruning for Model Compression
• Sparsifying Neural Network Connections for Face Recognition
• Neural Epitome Search for Architecture-Agnostic Network Compression
• BitNet: Bit-Regularized Deep Neural Networks
• Topology Optimization under Uncertainty using a Stochastic Gradient-based Approach
• Manipulating Identical Filter Redundancy for Efficient Pruning on Deep and Complicated CNN
• How Not to Give a FLOP: Combining Regularization and Pruning for Efficient Inference
• Finding trainable sparse networks through Neural Tangent Transfer
• Sketching and Neural Networks
• WSNet: Compact and Efficient Networks Through Weight Sampling
• DECORE: Deep Compression with Reinforcement Learning
• ESPN: Extremely Sparse Pruned Networks
• Smaller Models, Better Generalization
• Efficient architecture for deep neural networks with heterogeneous sensitivity
• Deep Asymmetric Networks with a Set of Node-wise Variant Activation Functions
• Disentangling Neural Architectures and Weights: A Case Study in Supervised Classification
• Building Compact and Robust Deep Neural Networks with Toeplitz Matrices
• DessiLBI: Exploring Structural Sparsity of Deep Networks via Differential Inclusion Paths
• HALO: Learning to Prune Neural Networks with Shrinkage
• Sum-Rate-Distortion Function for Indirect Multiterminal Source Coding in Federated Learning
• Differentiable Architecture Pruning for Transfer Learning
• The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation
• What needles do sparse neural networks find in nonlinear haystacks
• Recurrent Residual Module for Fast Inference in Videos
• $S^2$-LBI: Stochastic Split Linearized Bregman Iterations for Parsimonious Deep Learning
• Building Sparse Deep Feedforward Networks using Tree Receptive Fields
• Sparse Architectures for Text-Independent Speaker Verification Using Deep Neural Networks
• Training compact deep learning models for video classification using circulant matrices