Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization

Abstract

Optimization of Binarized Neural Networks (BNNs) currently relies on real-valued latent weights to accumulate small update steps. In this paper, we argue that these latent weights cannot be treated analogously to weights in real-valued networks. Instead their main role is to provide inertia during training. We interpret current methods in terms of inertia and provide novel insights into the optimization of BNNs. We subsequently introduce the first optimizer specifically designed for BNNs, Binary Optimizer (Bop), and demonstrate its performance on CIFAR-10 and ImageNet. Together, the redefinition of latent weights as inertia and the introduction of Bop enable a better understanding of BNN optimization and open up the way for further improvements in training methodologies for BNNs. Code is available at: https://github.com/plumerai/rethinking-bnn-optimization

arXiv commentAccepted at 33rd Conference on Neural Information Processing Systems (NeurIPS 2019), Vancouver, Canada - Updated ImageNet results

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Distilling the Knowledge in a Neural Network
• Decoupled Weight Decay Regularization
• Deep Residual Learning for Image Recognition
• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Deep Learning with Limited Numerical Precision
• Model compression via distillation and quantization
• Spatially-sparse convolutional neural networks
• Feature Learning in Deep Neural Networks - Studies on Speech Recognition Tasks
• Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy
• Deep neural networks are robust to weight binarization and other non-linear distortions
• Back to Simplicity: How to Train Accurate BNNs from Scratch?
• Probabilistic Binary Neural Networks
• Improved training of binary networks for human pose estimation and image recognition
• Structured Binary Neural Networks for Accurate Image Classification and Semantic Segmentation

Cited by in corpus (16)

• Synaptic metaplasticity in binarized neural networks
• Rotated Binary Neural Network
• ShiftAddNet: A Hardware-Inspired Deep Network
• Training Binary Neural Networks through Learning with Noisy Supervision
• ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions
• Training Binary Neural Networks using the Bayesian Learning Rule
• Mirror Descent View for Neural Network Quantization
• Learning Frequency Domain Approximation for Binary Neural Networks
• The Bayesian Learning Rule
• A Bop and Beyond: A Second Order Optimizer for Binarized Neural Networks
• High-Capacity Expert Binary Networks
• SiMaN: Sign-to-Magnitude Network Binarization
• ReCU: Reviving the Dead Weights in Binary Neural Networks
• "BNN - BN = ?": Training Binary Neural Networks without Batch Normalization
• Enabling Binary Neural Network Training on the Edge
• SmartDeal: Re-Modeling Deep Network Weights for Efficient Inference and Training