WoodFisher: Efficient Second-Order Approximation for Neural Network Compression

Abstract

Second-order information, in the form of Hessian- or Inverse-Hessian-vector products, is a fundamental tool for solving optimization problems. Recently, there has been significant interest in utilizing this information in the context of deep neural networks; however, relatively little is known about the quality of existing approximations in this context. Our work examines this question, identifies issues with existing approaches, and proposes a method called WoodFisher to compute a faithful and efficient estimate of the inverse Hessian. Our main application is to neural network compression, where we build on the classic Optimal Brain Damage/Surgeon framework. We demonstrate that WoodFisher significantly outperforms popular state-of-the-art methods for one-shot pruning. Further, even when iterative, gradual pruning is considered, our method results in a gain in test accuracy over the state-of-the-art approaches, for pruning popular neural networks (like ResNet-50, MobileNetV1) trained on standard image classification datasets such as ImageNet ILSVRC. We examine how our method can be extended to take into account first-order information, as well as illustrate its ability to automatically set layer-wise pruning thresholds and perform compression in the limited-data regime. The code is available at the following link, https://github.com/IST-DASLab/WoodFisher.

arXiv commentNeurIPS 2020

References in corpus (18)

• Adam: A Method for Stochastic Optimization
• Deep Learning in Neural Networks: An Overview
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Overcoming catastrophic forgetting in neural networks
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Understanding Black-box Predictions via Influence Functions
• To prune, or not to prune: exploring the efficacy of pruning for model compression
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• The State of Sparsity in Deep Neural Networks
• Sparse Networks from Scratch: Faster Training without Losing Performance
• Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization
• Rigging the Lottery: Making All Tickets Winners
• Soft Threshold Weight Reparameterization for Learnable Sparsity
• Dynamic Model Pruning with Feedback
• What is the State of Neural Network Pruning?
• Limitations of the Empirical Fisher Approximation for Natural Gradient Descent
• Model Fusion via Optimal Transport

Cited by in corpus (5)

• Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracy
• Revisiting Loss Modelling for Unstructured Pruning
• Neural Pruning via Growing Regularization
• Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition
• Deep Neural Compression Via Concurrent Pruning and Self-Distillation