Demystifying and Generalizing BinaryConnect

Abstract

BinaryConnect (BC) and its many variations have become the de facto standard for neural network quantization. However, our understanding of the inner workings of BC is still quite limited. We attempt to close this gap in four different aspects: (a) we show that existing quantization algorithms, including post-training quantization, are surprisingly similar to each other; (b) we argue for proximal maps as a natural family of quantizers that is both easy to design and analyze; (c) we refine the observation that BC is a special case of dual averaging, which itself is a special case of the generalized conditional gradient algorithm; (d) consequently, we propose ProxConnect (PC) as a generalization of BC and we prove its convergence properties by exploiting the established connections. We conduct experiments on CIFAR-10 and ImageNet, and verify that PC achieves competitive performance.

arXiv commentNeurIPS 2021

References in corpus (12)

• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Ternary Weight Networks
• Binary Neural Networks: A Survey
• A Survey on Methods and Theories of Quantized Neural Networks
• Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM
• Training Binary Neural Networks with Real-to-Binary Convolutions
• Characterising Bias in Compressed Models
• Probabilistic Binary Neural Networks
• On the Universal Approximability and Complexity Bounds of Quantized ReLU Neural Networks
• The High-Dimensional Geometry of Binary Neural Networks