ECC: Platform-Independent Energy-Constrained Deep Neural Network Compression via a Bilinear Regression Model

Abstract

Many DNN-enabled vision applications constantly operate under severe energy constraints such as unmanned aerial vehicles, Augmented Reality headsets, and smartphones. Designing DNNs that can meet a stringent energy budget is becoming increasingly important. This paper proposes ECC, a framework that compresses DNNs to meet a given energy constraint while minimizing accuracy loss. The key idea of ECC is to model the DNN energy consumption via a novel bilinear regression function. The energy estimate model allows us to formulate DNN compression as a constrained optimization that minimizes the DNN loss function over the energy constraint. The optimization problem, however, has nontrivial constraints. Therefore, existing deep learning solvers do not apply directly. We propose an optimization algorithm that combines the essence of the Alternating Direction Method of Multipliers (ADMM) framework with gradient-based learning algorithms. The algorithm decomposes the original constrained optimization into several subproblems that are solved iteratively and efficiently. ECC is also portable across different hardware platforms without requiring hardware knowledge. Experiments show that ECC achieves higher accuracy under the same or lower energy budget compared to state-of-the-art resource-constrained DNN compression techniques.

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• ADADELTA: An Adaptive Learning Rate Method
• Do Deep Nets Really Need to be Deep?
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Pruning Filters for Efficient ConvNets
• Stochastic Gradient Descent for Non-smooth Optimization: Convergence Results and Optimal Averaging Schemes
• Bayesian Compression for Deep Learning
• Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
• Structured Bayesian Pruning via Log-Normal Multiplicative Noise
• Compressing Neural Networks using the Variational Information Bottleneck
• A Unified Framework of DNN Weight Pruning and Weight Clustering/Quantization Using ADMM
• Energy-Constrained Compression for Deep Neural Networks via Weighted Sparse Projection and Layer Input Masking
• StrassenNets: Deep Learning with a Multiplication Budget