Meta-Regularization: An Approach to Adaptive Choice of the Learning Rate in Gradient Descent

Abstract

We propose \textit{Meta-Regularization}, a novel approach for the adaptive choice of the learning rate in first-order gradient descent methods. Our approach modifies the objective function by adding a regularization term on the learning rate, and casts the joint updating process of parameters and learning rates into a maxmin problem. Given any regularization term, our approach facilitates the generation of practical algorithms. When \textit{Meta-Regularization} takes the $\phi$-divergence as a regularizer, the resulting algorithms exhibit comparable theoretical convergence performance with other first-order gradient-based algorithms. Furthermore, we theoretically prove that some well-designed regularizers can improve the convergence performance under the strong-convexity condition of the objective function. Numerical experiments on benchmark problems demonstrate the effectiveness of algorithms derived from some common $\phi$-divergence in full batch as well as online learning settings.

References in corpus (5)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Optimal Distributed Online Prediction using Mini-Batches
• Online Learning Rate Adaptation with Hypergradient Descent
• A Unified Approach to Adaptive Regularization in Online and Stochastic Optimization
• Almost sure convergence and asymptotical normality of a generalization of Kesten's stochastic approximation algorithm for multidimensional case