A simpler approach to obtaining an O(1/t) convergence rate for the projected stochastic subgradient method
arXiv:1212.2002
Abstract
In this note, we present a new averaging technique for the projected stochastic subgradient method. By using a weighted average with a weight of t+1 for each iterate w_t at iteration t, we obtain the convergence rate of O(1/t) with both an easy proof and an easy implementation. The new scheme is compared empirically to existing techniques, with similar performance behavior.
8 pages, 6 figures. Changes with previous version: Added reference to concurrently submitted work arXiv:1212.1824v1; clarifications added; typos corrected; title changed to 'subgradient method' as 'subgradient descent' is misnomer
Cited by in corpus (6)
- On Analog Gradient Descent Learning over Multiple Access Fading Channels
- Guided parallelized stochastic gradient descent for delay compensation
- The Strength of Nesterov's Extrapolation in the Individual Convergence of Nonsmooth Optimization
- On Adaptive Stochastic Optimization for Streaming Data: A Newton's Method with O(dN) Operations
- Subgradient methods for non-smooth optimization problems with some relaxation of sharp minimum
- Distributed Computation of Stochastic GNE with Partial Information: An Augmented Best-Response Approach