Neural network relief: a pruning algorithm based on neural activity

Abstract

Current deep neural networks (DNNs) are overparameterized and use most of their neuronal connections during inference for each task. The human brain, however, developed specialized regions for different tasks and performs inference with a small fraction of its neuronal connections. We propose an iterative pruning strategy introducing a simple importance-score metric that deactivates unimportant connections, tackling overparameterization in DNNs and modulating the firing patterns. The aim is to find the smallest number of connections that is still capable of solving a given task with comparable accuracy, i.e. a simpler subnetwork. We achieve comparable performance for LeNet architectures on MNIST, and significantly higher parameter compression than state-of-the-art algorithms for VGG and ResNet architectures on CIFAR-10/100 and Tiny-ImageNet. Our approach also performs well for the two different optimizers considered -- Adam and SGD. The algorithm is not designed to minimize FLOPs when considering current hardware and software implementations, although it performs reasonably when compared to the state of the art.

References in corpus (15)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Bayesian Compression for Deep Learning
• Variational Dropout Sparsifies Deep Neural Networks
• Comparing Rewinding and Fine-tuning in Neural Network Pruning
• Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers
• Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon
• Net-Trim: Convex Pruning of Deep Neural Networks with Performance Guarantee
• CHIP: CHannel Independence-based Pruning for Compact Neural Networks
• Achieving Adversarial Robustness via Sparsity
• Shapley Value as Principled Metric for Structured Network Pruning