106 citations · 525 across the 44 of their papers we have counts for
5 papers · 1 filter
Adam: A Stochastic Method with Adaptive Variance Reduction
Mingrui Liu, Wei Zhang, Francesco Orabona +1
Adam is a widely used stochastic optimization method for deep learning applications. While practitioners prefer Adam because it requires less parameter tuning, its use is problemat…
A High Probability Analysis of Adaptive SGD with Momentum
Xiaoyu Li, Francesco Orabona
Stochastic Gradient Descent (SGD) and its variants are the most used algorithms in machine learning applications. In particular, SGD with adaptive learning rates and momentum is th…
Better Parameter-free Stochastic Optimization with ODE Updates for Coin-Betting
Keyi Chen, John Langford, Francesco Orabona
Parameter-free stochastic gradient descent (PFSGD) algorithms do not require setting learning rates while achieving optimal theoretical performance. In practical applications, howe…
Temporal Variability in Implicit Online Learning
Nicolò Campolongo, Francesco Orabona
In the setting of online learning, Implicit algorithms turn out to be highly successful from a practical standpoint. However, the tightest regret analyses only show marginal improv…
A Second look at Exponential and Cosine Step Sizes: Simplicity, Adaptivity, and Performance
Xiaoyu Li, Zhenxun Zhuang, Francesco Orabona
Stochastic Gradient Descent (SGD) is a popular tool in training large-scale machine learning models. Its performance, however, is highly variable, depending crucially on the choice…