WSNet: Compact and Efficient Networks Through Weight Sampling
arXiv:1711.10067
Abstract
We present a new approach and a novel architecture, termed WSNet, for learning compact and efficient deep neural networks. Existing approaches conventionally learn full model parameters independently and then compress them via ad hoc processing such as model pruning or filter factorization. Alternatively, WSNet proposes learning model parameters by sampling from a compact set of learnable parameters, which naturally enforces {parameter sharing} throughout the learning process. We demonstrate that such a novel weight sampling approach (and induced WSNet) promotes both weights and computation sharing favorably. By employing this method, we can more efficiently learn much smaller networks with competitive performance compared to baseline networks with equal numbers of convolution filters. Specifically, we consider learning compact and efficient 1D convolutional neural networks for audio classification. Extensive experiments on multiple audio classification datasets verify the effectiveness of WSNet. Combined with weight quantization, the resulted models are up to 180 times smaller and theoretically up to 16 times faster than the well-established baselines, without noticeable performance drop.
arXiv commentTo appear at ICML 2018
References in corpus (16)
- Adam: A Method for Stochastic Optimization
- Distilling the Knowledge in a Neural Network
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
- Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
- Compressing Deep Convolutional Networks using Vector Quantization
- Speeding up Convolutional Neural Networks with Low Rank Expansions
- Acoustic Scene Classification
- Xception: Deep Learning with Depthwise Separable Convolutions
- Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
- Fast Training of Convolutional Networks through FFTs
- Flattened Convolutional Neural Networks for Feedforward Acceleration
- Memory Bounded Deep Convolutional Networks
- Training Skinny Deep Neural Networks with Iterative Hard Thresholding Methods
- Dual Path Networks
- LCNN: Lookup-based Convolutional Neural Network