Random Bias Initialization Improves Quantized Training
arXiv:1909.13446
Abstract
Binary neural networks improve computationally efficiency of deep models with a large margin. However, there is still a performance gap between a successful full-precision training and binary training. We bring some insights about why this accuracy drop exists and call for a better understanding of binary network geometry. We start with analyzing full-precision neural networks with ReLU activation and compare it with its binarized version. This comparison suggests to initialize networks with random bias, a counter-intuitive remedy.
References in corpus (7)
- Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
- Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
- On the Number of Linear Regions of Deep Neural Networks
- Searching for Activation Functions
- Revise Saturated Activation Functions
- How Does Batch Normalization Help Binary Training?
- Improving LSTM-CTC based ASR performance in domains with limited training data