Switchable Precision Neural Networks

Abstract

Instantaneous and on demand accuracy-efficiency trade-off has been recently explored in the context of neural networks slimming. In this paper, we propose a flexible quantization strategy, termed Switchable Precision neural Networks (SP-Nets), to train a shared network capable of operating at multiple quantization levels. At runtime, the network can adjust its precision on the fly according to instant memory, latency, power consumption and accuracy demands. For example, by constraining the network weights to 1-bit with switchable precision activations, our shared network spans from BinaryConnect to Binarized Neural Network, allowing to perform dot-products using only summations or bit operations. In addition, a self-distillation scheme is proposed to increase the performance of the quantized switches. We tested our approach with three different quantizers and demonstrate the performance of SP-Nets against independently trained quantized models in classification accuracy for Tiny ImageNet and ImageNet datasets using ResNet-18 and MobileNet architectures.

References in corpus (6)

• Distilling the Knowledge in a Neural Network
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Slimmable Neural Networks
• AutoSlim: Towards One-Shot Architecture Search for Channel Numbers
• Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation
• Knowledge Adaptation for Efficient Semantic Segmentation