Explore the Knowledge contained in Network Weights to Obtain Sparse Neural Networks
arXiv:2103.15590
Abstract
Sparse neural networks are important for achieving better generalization and enhancing computation efficiency. This paper proposes a novel learning approach to obtain sparse fully connected layers in neural networks (NNs) automatically. We design a switcher neural network (SNN) to optimize the structure of the task neural network (TNN). The SNN takes the weights of the TNN as the inputs and its outputs are used to switch the connections of TNN. In this way, the knowledge contained in the weights of TNN is explored to determine the importance of each connection and the structure of TNN consequently. The SNN and TNN are learned alternately with stochastic gradient descent (SGD) optimization, targeting at a common objective. After learning, we achieve the optimal structure and the optimal parameters of the TNN simultaneously. In order to evaluate the proposed approach, we conduct image classification experiments on various network structures and datasets. The network structures include LeNet, ResNet18, ResNet34, VggNet16 and MobileNet. The datasets include MNIST, CIFAR10 and CIFAR100. The experimental results show that our approach can stably lead to sparse and well-performing fully connected layers in NNs.
References in corpus (12)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Distilling the Knowledge in a Neural Network
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Bayesian Compression for Deep Learning
- Variational Dropout Sparsifies Deep Neural Networks
- Structured Bayesian Pruning via Log-Normal Multiplicative Noise
- Compressing Neural Networks using the Variational Information Bottleneck
- N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning
- Learning Sparse Networks Using Targeted Dropout
- Training Skinny Deep Neural Networks with Iterative Hard Thresholding Methods
- Discovering Neural Wirings
- Diversity Networks: Neural Network Compression Using Determinantal Point Processes