Model compression as constrained optimization, with application to neural nets. Part I: general framework

Abstract

Compressing neural nets is an active research problem, given the large size of state-of-the-art nets for tasks such as object recognition, and the computational limits imposed by mobile devices. We give a general formulation of model compression as constrained optimization. This includes many types of compression: quantization, low-rank decomposition, pruning, lossless compression and others. Then, we give a general algorithm to optimize this nonconvex problem based on the augmented Lagrangian and alternating optimization. This results in a "learning-compression" algorithm, which alternates a learning step of the uncompressed model, independent of the compression type, with a compression step of the model parameters, independent of the learning task. This simple, efficient algorithm is guaranteed to find the best compressed model for the task in a local sense under standard assumptions. We present separately in several companion papers the development of this general framework into specific algorithms for model compression based on quantization, pruning and other variations, including experimental results on compressing neural nets and other models.

23 pages, 2 figures

References in corpus (11)

• Distilling the Knowledge in a Neural Network
• BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Compressing Deep Convolutional Networks using Vector Quantization
• Ternary Weight Networks
• Predicting Parameters in Deep Learning
• Tensorizing Neural Networks
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Distributed optimization of deeply nested systems
• Model compression as constrained optimization, with application to neural nets. Part II: quantization

Cited by in corpus (11)

• Massive MIMO is a Reality -- What is Next? Five Promising Research Directions for Antenna Arrays
• Model compression as constrained optimization, with application to neural nets. Part II: quantization
• A Programmable Approach to Neural Network Compression
• BinaryRelax: A Relaxation Approach For Training Deep Neural Networks With Quantized Weights
• A flexible, extensible software framework for model compression based on the LC algorithm
• Model compression as constrained optimization, with application to neural nets. Part V: combining compressions
• Generalized Ternary Connect: End-to-End Learning and Compression of Multiplication-Free Deep Neural Networks
• Going Beyond Classification Accuracy Metrics in Model Compression
• Search-Optimized Quantization in Biomedical Ontology Alignment
• Channel-wise pruning of neural networks with tapering resource constraint
• When in Doubt, Summon the Titans: Efficient Inference with Large Models