$S^3$: Sign-Sparse-Shift Reparametrization for Effective Training of Low-bit Shift Networks

Abstract

Shift neural networks reduce computation complexity by removing expensive multiplication operations and quantizing continuous weights into low-bit discrete values, which are fast and energy efficient compared to conventional neural networks. However, existing shift networks are sensitive to the weight initialization, and also yield a degraded performance caused by vanishing gradient and weight sign freezing problem. To address these issues, we propose S low-bit re-parameterization, a novel technique for training low-bit shift networks. Our method decomposes a discrete parameter in a sign-sparse-shift 3-fold manner. In this way, it efficiently learns a low-bit network with a weight dynamics similar to full-precision networks and insensitive to weight initialization. Our proposed training method pushes the boundaries of shift neural networks and shows 3-bit shift networks out-performs their full-precision counterparts in terms of top-1 accuracy on ImageNet.

References in corpus (9)

• Trained Ternary Quantization
• Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks
• ShiftCNN: Generalized Low-Precision Architecture for Inference of Convolutional Neural Networks
• ShiftAddNet: A Hardware-Inspired Deep Network
• Searching for Low-Bit Weights in Quantized Neural Networks
• The Difficulty of Training Sparse Neural Networks
• Kernel Based Progressive Distillation for Adder Neural Networks
• Learning Accurate Low-Bit Deep Neural Networks with Stochastic Quantization
• AdderNet and its Minimalist Hardware Design for Energy-Efficient Artificial Intelligence