Adaptive Hierarchical Hyper-gradient Descent
arXiv:2008.07277
Abstract
In this study, we investigate learning rate adaption at different levels based on the hyper-gradient descent framework and propose a method that adaptively learns the optimizer parameters by combining multiple levels of learning rates with hierarchical structures. Meanwhile, we show the relationship between regularizing over-parameterized learning rates and building combinations of adaptive learning rates at different levels. The experiments on several network architectures, including feed-forward networks, LeNet-5 and ResNet-18/34, show that the proposed multi-level adaptive approach can outperform baseline adaptive methods in a variety of circumstances.
References in corpus (10)
- An overview of gradient descent optimization algorithms
- Improved Regularization of Convolutional Neural Networks with Cutout
- On the Convergence of Adam and Beyond
- Adaptive Gradient Methods with Dynamic Bound of Learning Rate
- Optimization for deep learning: theory and algorithms
- Learned Optimizers that Scale and Generalize
- Online Learning Rate Adaptation with Hypergradient Descent
- Learning Gradient Descent: Better Generalization and Longer Horizons
- On the Convergence of AdaBound and its Connection to SGD
- Using Statistics to Automate Stochastic Optimization