DTN: A Learning Rate Scheme with Convergence Rate of $\mathcal{O}(1/t)$ for SGD

Abstract

This paper has some inconsistent results, i.e., we made some failed claims because we did some mistakes for using the test criterion for a series. Precisely, our claims on the convergence rate of $\mathcal{O}(1/t)$ of SGD presented in Theorem 1, Corollary 1, Theorem 2 and Corollary 2 are wrongly derived because they are based on Lemma 5. In Lemma 5, we do not correctly use the test criterion for a series. Hence, the result of Lemma 5 is not valid. We would like to thank the community for pointing out this mistake!

arXiv commentThis paper has inconsistent results, i.e., we made some failed claims because we did some mistakes for using the test criterion for a series

References in corpus (11)

• ADADELTA: An Adaptive Learning Rate Method
• SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives
• Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networks
• Optimization with First-Order Surrogate Functions
• Stochastic Recursive Gradient Algorithm for Nonconvex Optimization
• Natasha: Faster Non-Convex Stochastic Optimization Via Strongly Non-Convex Parameter
• Theoretical properties of the global optimizer of two layer neural network
• Fast and Faster Convergence of SGD for Over-Parameterized Models and an Accelerated Perceptron
• New Convergence Aspects of Stochastic Gradient Algorithms
• When Does Stochastic Gradient Algorithm Work Well?
• Characterization of Convex Objective Functions and Optimal Expected Convergence Rates for SGD