Training Binary Neural Networks through Learning with Noisy Supervision

Abstract

This paper formalizes the binarization operations over neural networks from a learning perspective. In contrast to classical hand crafted rules (\eg hard thresholding) to binarize full-precision neurons, we propose to learn a mapping from full-precision neurons to the target binary ones. Each individual weight entry will not be binarized independently. Instead, they are taken as a whole to accomplish the binarization, just as they work together in generating convolution features. To help the training of the binarization mapping, the full-precision neurons after taking sign operations is regarded as some auxiliary supervision signal, which is noisy but still has valuable guidance. An unbiased estimator is therefore introduced to mitigate the influence of the supervision noise. Experimental results on benchmark datasets indicate that the proposed binarization technique attains consistent improvements over baselines.

arXiv commentICML 2020

References in corpus (17)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• Binarized Neural Networks
• Regularizing Neural Networks by Penalizing Confident Output Distributions
• Masking: A New Perspective of Noisy Supervision
• How does Disagreement Help Generalization against Label Corruption?
• Symmetric Cross Entropy for Robust Learning with Noisy Labels
• Joint Optimization Framework for Learning with Noisy Labels
• Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization
• Deep Learning with Low Precision by Half-wave Gaussian Quantization
• Iterative Learning with Open-set Noisy Labels
• GBCNs: Genetic Binary Convolutional Networks for Enhancing the Performance of 1-bit DCNNs

Cited by in corpus (12)

• GhostNets on Heterogeneous Devices via Cheap Operations
• SCOP: Scientific Control for Reliable Neural Network Pruning
• Lightweight Pixel Difference Networks for Efficient Visual Representation Learning
• Model Rubik's Cube: Twisting Resolution, Depth and Width for TinyNets
• Kernel Based Progressive Distillation for Adder Neural Networks
• Mirror Descent View for Neural Network Quantization
• Patch Slimming for Efficient Vision Transformers
• Learning Frequency Domain Approximation for Binary Neural Networks
• Learning Versatile Convolution Filters for Efficient Visual Recognition
• Learning with Noisy Labels by Efficient Transition Matrix Estimation to Combat Label Miscorrection
• SiMaN: Sign-to-Magnitude Network Binarization
• Unsupervised Representation Learning for Binary Networks by Joint Classifier Learning