Forget the Learning Rate, Decay Loss
arXiv:1905.00094
Abstract
In the usual deep neural network optimization process, the learning rate is the most important hyper parameter, which greatly affects the final convergence effect. The purpose of learning rate is to control the stepsize and gradually reduce the impact of noise on the network. In this paper, we will use a fixed learning rate with method of decaying loss to control the magnitude of the update. We used Image classification, Semantic segmentation, and GANs to verify this method. Experiments show that the loss decay strategy can greatly improve the performance of the model
arXiv commentAsia Conference on Machine Learning and Computing. arXiv admin note: text overlap with arXiv:1703.04782, arXiv:1611.07004 by other authors
References in corpus (7)
- TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
- Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
- MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
- Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
- Improved Techniques for Training GANs
- Energy-based Generative Adversarial Network
- Fast, Better Training Trick -- Random Gradient