NoiseOut: A Simple Way to Prune Neural Networks

Abstract

Neural networks are usually over-parameterized with significant redundancy in the number of required neurons which results in unnecessary computation and memory usage at inference time. One common approach to address this issue is to prune these big networks by removing extra neurons and parameters while maintaining the accuracy. In this paper, we propose NoiseOut, a fully automated pruning algorithm based on the correlation between activations of neurons in the hidden layers. We prove that adding additional output neurons with entirely random targets results into a higher correlation between neurons which makes pruning by NoiseOut even more efficient. Finally, we test our method on various networks and datasets. These experiments exhibit high pruning rates while maintaining the accuracy of the original network.

References in corpus (1)

• An exploration of parameter redundancy in deep networks with circulant projections

Cited by in corpus (13)

• Compression-aware Training of Deep Networks
• Learning Sparse Networks Using Targeted Dropout
• Understanding Neural Networks and Individual Neuron Importance via Information-Ordered Cumulative Ablation
• EffNet: An Efficient Structure for Convolutional Neural Networks
• Full deep neural network training on a pruned weight budget
• Artificial neural networks condensation: A strategy to facilitate adaption of machine learning in medical settings by reducing computational burden
• Learning Neural Network Classifiers with Low Model Complexity
• Smaller Models, Better Generalization
• MPDCompress - Matrix Permutation Decomposition Algorithm for Deep Neural Network Compression
• Neural Machine Translation: A Review and Survey
• Using noise to probe recurrent neural network structure and prune synapses
• Adaptive Ensemble of Classifiers with Regularization for Imbalanced Data Classification
• Solving hybrid machine learning tasks by traversing weight space geodesics