Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM

Abstract

Although deep learning models are highly effective for various learning tasks, their high computational costs prohibit the deployment to scenarios where either memory or computational resources are limited. In this paper, we focus on compressing and accelerating deep models with network weights represented by very small numbers of bits, referred to as extremely low bit neural network. We model this problem as a discretely constrained optimization problem. Borrowing the idea from Alternating Direction Method of Multipliers (ADMM), we decouple the continuous parameters from the discrete constraints of network, and cast the original hard problem into several subproblems. We propose to solve these subproblems using extragradient and iterative quantization algorithms that lead to considerably faster convergency compared to conventional optimization methods. Extensive experiments on image recognition and object detection verify that the proposed algorithm is more effective than state-of-the-art approaches when coming to extremely low bit neural network.

Cited by in corpus (45)

• Binary Neural Networks: A Survey
• Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluation
• Compact recurrent neural networks for acoustic event detection on low-energy low-complexity platforms
• Pruning and Quantization for Deep Neural Network Acceleration: A Survey
• AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Rates
• Towards Efficient Training for Neural Network Quantization
• AdaBits: Neural Network Quantization with Adaptive Bit-Widths
• Mixed Precision Low-bit Quantization of Neural Network Language Models for Speech Recognition
• Comprehensive SNN Compression Using ADMM Optimization and Activity Regularization
• CGaP: Continuous Growth and Pruning for Efficient Deep Learning
• Weight Normalization based Quantization for Deep Neural Network Compression
• TernaryBERT: Distillation-aware Ultra-low Bit BERT
• Mirror Descent View for Neural Network Quantization
• RPR: Random Partition Relaxation for Training; Binary and Ternary Weight Neural Networks
• Elastic Significant Bit Quantization and Acceleration for Deep Neural Networks
• Mix and Match: A Novel FPGA-Centric Deep Neural Network Quantization Framework
• Automatic Neural Network Compression by Sparsity-Quantization Joint Learning: A Constrained Optimization-based Approach
• Model compression as constrained optimization, with application to neural nets. Part V: combining compressions
• MSP: An FPGA-Specific Mixed-Scheme, Multi-Precision Deep Neural Network Quantization Framework
• Binarized Neural Architecture Search for Efficient Object Recognition
• Alternating Direction Method of Multipliers for Quantization
• A Unified DNN Weight Compression Framework Using Reweighted Optimization Methods
• Learning Sparse & Ternary Neural Networks with Entropy-Constrained Trained Ternarization (EC2T)
• Dynamic Network Quantization for Efficient Video Inference
• On the Convergence of (Stochastic) Gradient Descent with Extrapolation for Non-Convex Optimization
• ReCU: Reviving the Dead Weights in Binary Neural Networks
• SiMaN: Sign-to-Magnitude Network Binarization
• 3U-EdgeAI: Ultra-Low Memory Training, Ultra-Low BitwidthQuantization, and Ultra-Low Latency Acceleration
• Differentiable Feature Aggregation Search for Knowledge Distillation
• LANCE: Efficient Low-Precision Quantized Winograd Convolution for Neural Networks Based on Graphics Processing Units
• SQWA: Stochastic Quantized Weight Averaging for Improving the Generalization Capability of Low-Precision Deep Neural Networks
• Pruning Ternary Quantization
• Sparsity-Control Ternary Weight Networks
• GRIM: A General, Real-Time Deep Learning Inference Framework for Mobile Devices based on Fine-Grained Structured Weight Sparsity
• BAMSProd: A Step towards Generalizing the Adaptive Optimization Methods to Deep Binary Model
• Architecture Aware Latency Constrained Sparse Neural Networks
• Unified Signal Compression Using a GAN with Iterative Latent Representation Optimization
• Towards Modality Transferable Visual Information Representation with Optimal Model Compression
• Kernel Quantization for Efficient Network Compression
• Mixed Precision DNN Qunatization for Overlapped Speech Separation and Recognition
• CBP: Backpropagation with constraint on weight precision using a pseudo-Lagrange multiplier method
• Alternate Model Growth and Pruning for Efficient Training of Recommendation Systems
• Demystifying and Generalizing BinaryConnect
• RMSMP: A Novel Deep Neural Network Quantization Framework with Row-wise Mixed Schemes and Multiple Precisions
• Low-bit Quantization of Recurrent Neural Network Language Models Using Alternating Direction Methods of Multipliers