Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM

Abstract

Although deep learning models are highly effective for various learning tasks, their high computational costs prohibit the deployment to scenarios where either memory or computational resources are limited. In this paper, we focus on compressing and accelerating deep models with network weights represented by very small numbers of bits, referred to as extremely low bit neural network. We model this problem as a discretely constrained optimization problem. Borrowing the idea from Alternating Direction Method of Multipliers (ADMM), we decouple the continuous parameters from the discrete constraints of network, and cast the original hard problem into several subproblems. We propose to solve these subproblems using extragradient and iterative quantization algorithms that lead to considerably faster convergency compared to conventional optimization methods. Extensive experiments on image recognition and object detection verify that the proposed algorithm is more effective than state-of-the-art approaches when coming to extremely low bit neural network.

Cited by in corpus (13)

• Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation
• Compact recurrent neural networks for acoustic event detection on low-energy low-complexity platforms
• Towards Efficient Training for Neural Network Quantization
• CGaP: Continuous Growth and Pruning for Efficient Deep Learning
• Weight Normalization based Quantization for Deep Neural Network Compression
• RPR: Random Partition Relaxation for Training; Binary and Ternary Weight Neural Networks
• A Unified DNN Weight Compression Framework Using Reweighted Optimization Methods
• Binarized Neural Architecture Search for Efficient Object Recognition
• On the Convergence of (Stochastic) Gradient Descent with Extrapolation for Non-Convex Optimization
• SQWA: Stochastic Quantized Weight Averaging for Improving the Generalization Capability of Low-Precision Deep Neural Networks
• Differentiable Feature Aggregation Search for Knowledge Distillation
• Towards Modality Transferable Visual Information Representation with Optimal Model Compression
• BAMSProd: A Step towards Generalizing the Adaptive Optimization Methods to Deep Binary Model