RPR: Random Partition Relaxation for Training; Binary and Ternary Weight Neural Networks
arXiv:2001.01091
Abstract
We present Random Partition Relaxation (RPR), a method for strong quantization of neural networks weight to binary (+1/-1) and ternary (+1/0/-1) values. Starting from a pre-trained model, we quantize the weights and then relax random partitions of them to their continuous values for retraining before re-quantizing them and switching to another weight partition for further adaptation. We demonstrate binary and ternary-weight networks with accuracies beyond the state-of-the-art for GoogLeNet and competitive performance for ResNet-18 and ResNet-50 using an SGD-based training method that can easily be integrated into existing frameworks.
References in corpus (8)
- Ternary Weight Networks
- Origami: A 803 GOp/s/W Convolutional Network Accelerator
- Improving Neural Network Quantization without Retraining using Outlier Channel Splitting
- FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search
- Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM
- HAQ: Hardware-Aware Automated Quantization with Mixed Precision
- HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision
- FANN-on-MCU: An Open-Source Toolkit for Energy-Efficient Neural Network Inference at the Edge of the Internet of Things