A Programmable Approach to Neural Network Compression

Abstract

Deep neural networks (DNNs) frequently contain far more weights, represented at a higher precision, than are required for the specific task which they are trained to perform. Consequently, they can often be compressed using techniques such as weight pruning and quantization that reduce both the model size and inference time without appreciable loss in accuracy. However, finding the best compression strategy and corresponding target sparsity for a given DNN, hardware platform, and optimization objective currently requires expensive, frequently manual, trial-and-error experimentation. In this paper, we introduce a programmable system for model compression called Condensa. Users programmatically compose simple operators, in Python, to build more complex and practically interesting compression strategies. Given a strategy and user-provided objective (such as minimization of running time), Condensa uses a novel Bayesian optimization-based algorithm to automatically infer desirable sparsities. Our experiments on four real-world DNNs demonstrate memory footprint and hardware runtime throughput improvements of 188x and 2.59x, respectively, using at most ten samples per search. We have released a reference implementation of Condensa at https://github.com/NVlabs/condensa.

arXiv commentThis is an updated version of a paper published in IEEE Micro, vol. 40, no. 5, pp. 17-25, Sept.-Oct. 2020 at https://ieeexplore.ieee.org/document/9151283

References in corpus (23)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Practical Bayesian Optimization of Machine Learning Algorithms
• Playing Atari with Deep Reinforcement Learning
• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Compressing Deep Convolutional Networks using Vector Quantization
• Trained Ternary Quantization
• Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• Pruning Filters for Efficient ConvNets
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Pointer Sentinel Mixture Models
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP
• CondenseNet: An Efficient DenseNet using Learned Group Convolutions
• Structured Pruning of a BERT-based Question Answering Model
• Compact Deep Convolutional Neural Networks With Coarse Pruning
• AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Rates
• Model compression as constrained optimization, with application to neural nets. Part I: general framework
• Asymptotic Soft Filter Pruning for Deep Convolutional Neural Networks

Cited by in corpus (5)

• Machine Learning at the Network Edge: A Survey
• HighLight: Efficient and Flexible DNN Acceleration with Hierarchical Structured Sparsity
• Fixed-Point Code Synthesis For Neural Networks
• Going Beyond Classification Accuracy Metrics in Model Compression
• MOFHEI: Model Optimizing Framework for Fast and Efficient Homomorphically Encrypted Neural Network Inference