Scaling Up Exact Neural Network Compression by ReLU Stability

Abstract

We can compress a rectifier network while exactly preserving its underlying functionality with respect to a given input domain if some of its neurons are stable. However, current approaches to determine the stability of neurons with Rectified Linear Unit (ReLU) activations require solving or finding a good approximation to multiple discrete optimization problems. In this work, we introduce an algorithm based on solving a single optimization problem to identify all stable neurons. Our approach is on median 183 times faster than the state-of-art method on CIFAR-10, which allows us to explore exact compression on deeper (5 x 100) and wider (2 x 800) networks within minutes. For classifiers trained under an amount of L1 regularization that does not worsen accuracy, we can remove up to 56% of the connections on the CIFAR-10 dataset. The code is available at the following link, https://github.com/yuxwind/ExactCompression.

arXiv commentNeurIPS 2021

References in corpus (23)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• Rethinking the Value of Network Pruning
• On the Number of Linear Regions of Deep Neural Networks
• SNIP: Single-shot Network Pruning based on Connection Sensitivity
• Evaluating Robustness of Neural Networks with Mixed Integer Programming
• Comparing Rewinding and Fine-tuning in Neural Network Pruning
• Picking Winning Tickets Before Training by Preserving Gradient Flow
• Training for Faster Adversarial Robustness Verification via Inducing ReLU Stability
• Deep ReLU Networks Have Surprisingly Few Activation Patterns
• Non-Vacuous Generalization Bounds at the ImageNet Scale: A PAC-Bayesian Compression Approach
• Complexity of Linear Regions in Deep Networks
• Diversity Networks: Neural Network Compression Using Determinantal Point Processes
• Characterising Bias in Compressed Models
• Algorithms for Verifying Deep Neural Networks
• EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis
• Tensorial Neural Networks: Generalization of Neural Networks and Application to Model Compression
• On the Predictability of Pruning Across Scales
• Understanding Generalization in Deep Learning via Tensor Methods
• Equivalent and Approximate Transformations of Deep Neural Networks
• Prune Responsibly

Cited by in corpus (1)

• Constrained Discrete Black-Box Optimization using Mixed-Integer Programming