Training Sparse Neural Networks
arXiv:1611.06694
Abstract
Deep neural networks with lots of parameters are typically used for large-scale computer vision tasks such as image classification. This is a result of using dense matrix multiplications and convolutions. However, sparse computations are known to be much more efficient. In this work, we train and build neural networks which implicitly use sparse computations. We introduce additional gate variables to perform parameter selection and show that this is equivalent to using a spike-and-slab prior. We experimentally validate our method on both small and large networks and achieve state-of-the-art compression results for sparse neural network models.
References in corpus (10)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
- Compressing Deep Convolutional Networks using Vector Quantization
- Predicting Parameters in Deep Learning
- Tensorizing Neural Networks
- Memory Bounded Deep Convolutional Networks
- An exploration of parameter redundancy in deep networks with circulant projections
- ACDC: A Structured Efficient Linear Layer
- Deep Fried Convnets
Cited by in corpus (7)
- Learning Efficient Convolutional Networks through Network Slimming
- Winning the Lottery with Continuous Sparsification
- Pruning at a Glance: Global Neural Pruning for Model Compression
- Combinatorial Designs for Deep Learning
- On the Robustness of Monte Carlo Dropout Trained with Noisy Labels
- Learning Sparsity of Representations with Discrete Latent Variables
- Architecture-aware Network Pruning for Vision Quality Applications