Rate Distortion For Model Compression: From Theory To Practice

Abstract

The enormous size of modern deep neural networks makes it challenging to deploy those models in memory and communication limited scenarios. Thus, compressing a trained model without a significant loss in performance has become an increasingly important task. Tremendous advances has been made recently, where the main technical building blocks are parameter pruning, parameter sharing (quantization), and low-rank factorization. In this paper, we propose principled approaches to improve upon the common heuristics used in those building blocks, namely pruning and quantization. We first study the fundamental limit for model compression via the rate distortion theory. We bring the rate distortion function from data compression to model compression to quantify this fundamental limit. We prove a lower bound for the rate distortion function and prove its achievability for linear models. Although this achievable compression scheme is intractable in practice, this analysis motivates a novel model compression framework. This framework provides a new objective function in model compression, which can be applied together with other classes of model compressor such as pruning or quantization. Theoretically, we prove that the proposed scheme is optimal for compressing one-hidden-layer ReLU neural networks. Empirically, we show that the proposed scheme improves upon the baseline in the compression-accuracy tradeoff.

23 pages, 12 figures

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• On Calibration of Modern Neural Networks
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Compressing Deep Convolutional Networks using Vector Quantization
• Compressing Neural Networks with the Hashing Trick
• Bayesian Compression for Deep Learning
• Learning One-hidden-layer Neural Networks with Landscape Design
• Towards the Limit of Network Quantization
• The Nearest Neighbor Information Estimator is Adaptively Near Minimax Rate-Optimal
• Improved Bayesian Compression

Cited by in corpus (8)

• Optimal Gradient Compression for Distributed and Federated Learning
• On Finite Blocklength Lossy Source Coding
• A Gradient Flow Framework For Analyzing Network Pruning
• Taxonomy and Evaluation of Structured Compression of Convolutional Neural Networks
• A Generalized Zero-Shot Quantization of Deep Convolutional Neural Networks via Learned Weights Statistics
• High-dimensional Asymptotics of VAEs: Threshold of Posterior Collapse and Dataset-Size Dependence of Rate-Distortion Curve
• Rate-Distortion Analysis of Minimum Excess Risk in Bayesian Learning
• Information-Theoretic Understanding of Population Risk Improvement with Model Compression