Pruning by Explaining: A Novel Criterion for Deep Neural Network Pruning

Abstract

The success of convolutional neural networks (CNNs) in various applications is accompanied by a significant increase in computation and parameter storage costs. Recent efforts to reduce these overheads involve pruning and compressing the weights of various layers while at the same time aiming to not sacrifice performance. In this paper, we propose a novel criterion for CNN pruning inspired by neural network interpretability: The most relevant units, i.e. weights or filters, are automatically found using their relevance scores obtained from concepts of explainable AI (XAI). By exploring this idea, we connect the lines of interpretability and model compression research. We show that our proposed method can efficiently prune CNN models in transfer-learning setups in which networks pre-trained on large corpora are adapted to specialized tasks. The method is evaluated on a broad range of computer vision datasets. Notably, our novel criterion is not only competitive or better compared to state-of-the-art pruning criteria when successive retraining is performed, but clearly outperforms these previous criteria in the resource-constrained application scenario in which the data of the task to be transferred to is very scarce and one chooses to refrain from fine-tuning. Our method is able to compress the model iteratively while maintaining or even improving accuracy. At the same time, it has a computational cost in the order of gradient computation and is comparatively simple to apply without the need for tuning hyperparameters for pruning.

arXiv comment25 pages + 5 supplementary pages, 13 figures, 6 tables

References in corpus (21)

• Scikit-learn: Machine Learning in Python
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Deep learning with convolutional neural networks for EEG decoding and visualization
• Methods for Interpreting and Understanding Deep Neural Networks
• Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Explaining NonLinear Classification Decisions with Deep Taylor Decomposition
• Recent Trends in Deep Learning Based Natural Language Processing
• Fine-Grained Visual Classification of Aircraft
• Unmasking Clever Hans Predictors and Assessing What Machines Really Learn
• ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices
• Benchmark Analysis of Representative Deep Neural Network Architectures
• Predicting Parameters in Deep Learning
• Channel Pruning for Accelerating Very Deep Neural Networks
• "What is Relevant in a Text Document?": An Interpretable Machine Learning Approach
• Resolving challenges in deep learning-based analyses of histopathological images using explanation methods
• EIE: Efficient Inference Engine on Compressed Deep Neural Network
• meProp: Sparsified Back Propagation for Accelerated Deep Learning with Reduced Overfitting
• Structured Probabilistic Pruning for Convolutional Neural Network Acceleration
• Lets keep it simple, Using simple architectures to outperform deeper and more complex architectures
• Breaking Batch Normalization for better explainability of Deep Neural Networks through Layer-wise Relevance Propagation

Cited by in corpus (23)

• Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directions
• Opportunities and Challenges in Explainable Artificial Intelligence (XAI): A Survey
• Ground Truth Evaluation of Neural Network Explanations with CLEVR-XAI
• Explainable, Domain-Adaptive, and Federated Artificial Intelligence in Medicine
• Pruning neural networks without any data by iteratively conserving synaptic flow
• Optimizing Convolutional Neural Network Architecture
• Disentangled Explanations of Neural Network Predictions by Finding Relevant Subspaces
• Taxonomy of Saliency Metrics for Channel Pruning
• Adversarial Pruning: A Survey and Benchmark of Pruning Methods for Adversarial Robustness
• A Novel Correlation-optimized Deep Learning Method for Wind Speed Forecast
• ECQ${}^{\text{x}}$: Explainability-Driven Quantization for Low-Bit and Sparse DNNs
• Toward Compact Deep Neural Networks via Energy-Aware Pruning
• Preemptively Pruning Clever-Hans Strategies in Deep Neural Networks
• Shapley Value as Principled Metric for Structured Network Pruning
• The Thousand Faces of Explainable AI Along the Machine Learning Life Cycle: Industrial Reality and Current State of Research
• Utilizing Explainable AI for Quantization and Pruning of Deep Neural Networks
• Trends and Advancements in Deep Neural Network Communication
• Automatic Neural Network Pruning that Efficiently Preserves the Model Accuracy
• Sparse Explanations of Neural Networks Using Pruned Layer-Wise Relevance Propagation
• Towards Explaining Deep Neural Network Compression Through a Probabilistic Latent Space
• ADDS: Adaptive Differentiable Sampling for Robust Multi-Party Learning
• HASTE: A Framework for Training-Free, Dynamic, and Steerable Compression of Pre-Trained Convolutional Neural Networks
• An accuracy-aware extension to LRP-based pruning for CNNs to prevent cascading accuracy degradation in data-scarce transfer learning