Learning Frequency Domain Approximation for Binary Neural Networks
arXiv:2103.00841
Abstract
Binary neural networks (BNNs) represent original full-precision weights and activations into 1-bit with sign function. Since the gradient of the conventional sign function is almost zero everywhere which cannot be used for back-propagation, several attempts have been proposed to alleviate the optimization difficulty by using approximate gradient. However, those approximations corrupt the main direction of factual gradient. To this end, we propose to estimate the gradient of sign function in the Fourier frequency domain using the combination of sine functions for training BNNs, namely frequency domain approximation (FDA). The proposed approach does not affect the low-frequency information of the original sign function which occupies most of the overall energy, and high-frequency coefficients will be ignored to avoid the huge computational overhead. In addition, we embed a noise adaptation module into the training phase to compensate the approximation error. The experiments on several benchmark datasets and neural architectures illustrate that the binary network learned using our method achieves the state-of-the-art accuracy. Code will be available at \textit{https://gitee.com/mindspore/models/tree/master/research/cv/FDA-BNN}.
arXiv comment12 pages
References in corpus (17)
- Distilling the Knowledge in a Neural Network
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
- Binarized Neural Networks
- Masking: A New Perspective of Noisy Supervision
- How does Disagreement Help Generalization against Label Corruption?
- XNOR-Net++: Improved Binary Neural Networks
- Latent Weights Do Not Exist: Rethinking Binarized Neural Network Optimization
- Rotated Binary Neural Network
- Back to Simplicity: How to Train Accurate BNNs from Scratch?
- Training Binary Neural Networks through Learning with Noisy Supervision
- Searching for Low-Bit Weights in Quantized Neural Networks
- Kernel Based Progressive Distillation for Adder Neural Networks
- Towards Frequency-Based Explanation for Robust CNN
- Positive-Unlabeled Compression on the Cloud
- Towards Understanding the Importance of Shortcut Connections in Residual Networks
- Dynamic Resolution Network