EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis

Abstract

Reducing the test time resource requirements of a neural network while preserving test accuracy is crucial for running inference on resource-constrained devices. To achieve this goal, we introduce a novel network reparameterization based on the Kronecker-factored eigenbasis (KFE), and then apply Hessian-based structured pruning methods in this basis. As opposed to existing Hessian-based pruning algorithms which do pruning in parameter coordinates, our method works in the KFE where different weights are approximately independent, enabling accurate pruning and fast computation. We demonstrate empirically the effectiveness of the proposed method through extensive experiments. In particular, we highlight that the improvements are especially significant for more challenging datasets and networks. With negligible loss of accuracy, an iterative-pruning version gives a 10$\times$ reduction in model size and a 8$\times$ reduction in FLOPs on wide ResNet32.

ICML 2019

Cited by in corpus (18)

• Picking Winning Tickets Before Training by Preserving Gradient Flow
• Applications and Techniques for Fast Machine Learning in Science
• End-to-End Supermask Pruning: Learning to Prune Image Captioning Models
• A Brain-inspired Algorithm for Training Highly Sparse Neural Networks
• Single Shot Structured Pruning Before Training
• Which Algorithmic Choices Matter at Which Batch Sizes? Insights From a Noisy Quadratic Model
• Convolutional Neural Network Pruning with Structural Redundancy Reduction
• Dissecting Hessian: Understanding Common Structure of Hessian in Neural Networks
• Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra
• Filter Sketch for Network Pruning
• Scaling Up Exact Neural Network Compression by ReLU Stability
• Dynamical Isometry: The Missing Ingredient for Neural Network Pruning
• Neural Pruning via Growing Regularization
• Deep Neural Compression Via Concurrent Pruning and Self-Distillation
• Initialization and Regularization of Factorized Neural Layers
• SOSP: Efficiently Capturing Global Correlations by Second-Order Structured Pruning
• Dissecting Non-Vacuous Generalization Bounds based on the Mean-Field Approximation
• How Well Do Sparse Imagenet Models Transfer?