106 citations · 525 across the 44 of their papers we have counts for
34 papers · 1 filter
Normalized Gradients for All
Francesco Orabona
In this short note, I show how to adapt to Hölder smoothness using normalized gradients in a black-box way. Moreover, the bound will depend on a novel notion of local Hölder smooth…
Implicit Interpretation of Importance Weight Aware Updates
Keyi Chen, Francesco Orabona
Due to its speed and simplicity, subgradient descent is one of the most used optimization algorithms in convex machine learning algorithms. However, tuning its learning rate is pro…
Generalized Implicit Follow-The-Regularized-Leader
Keyi Chen, Francesco Orabona
We propose a new class of online learning algorithms, generalized implicit Follow-The-Regularized-Leader (FTRL), that expands the scope of FTRL framework. Generalized implicit FTRL…
Tighter PAC-Bayes Bounds Through Coin-Betting
Kyoungseok Jang, Kwang-Sung Jun, Ilja Kuzborskij +1
We consider the problem of estimating the mean of a sequence of random elements where is a fixed scalar function, ar…
Optimal Stochastic Non-smooth Non-convex Optimization through Online-to-Non-convex Conversion
Ashok Cutkosky, Harsh Mehta, Francesco Orabona
We present new algorithms for optimizing non-smooth, non-convex stochastic objectives based on a novel analysis technique. This improves the current best-known complexity for findi…
Robustness to Unbounded Smoothness of Generalized SignSGD
Michael Crawshaw, Mingrui Liu, Francesco Orabona +2
Traditional analyses in non-convex optimization typically rely on the smoothness assumption, namely requiring the gradients to be Lipschitz. However, recent evidence shows that thi…
Implicit Parameter-free Online Learning with Truncated Linear Models
Keyi Chen, Ashok Cutkosky, Francesco Orabona
Parameter-free algorithms are online learning algorithms that do not require setting learning rates. They achieve optimal regret with respect to the distance between the initial po…
Understanding AdamW through Proximal Methods and Scale-Freeness
Zhenxun Zhuang, Mingrui Liu, Ashok Cutkosky +1
Adam has been widely adopted for training deep neural networks due to less hyperparameter tuning and remarkable performance. To improve generalization, Adam is typically used in ta…
Online Learning with Optimism and Delay
Genevieve Flaspohler, Francesco Orabona, Judah Cohen +4
Inspired by the demands of real-time climate and weather forecasting, we develop optimistic online learning algorithms that require no parameter tuning and have optimal regret guar…
A closer look at temporal variability in dynamic online learning
Nicolò Campolongo, Francesco Orabona
This work focuses on the setting of dynamic regret in the context of online learning with full information. In particular, we analyze regret bounds with respect to the temporal var…
On the Last Iterate Convergence of Momentum Methods
Xiaoyu Li, Mingrui Liu, Francesco Orabona
SGD with Momentum (SGDM) is a widely used family of algorithms for large-scale optimization of machine learning problems. Yet, when optimizing generic convex functions, no advantag…
Adam: A Stochastic Method with Adaptive Variance Reduction
Mingrui Liu, Wei Zhang, Francesco Orabona +1
Adam is a widely used stochastic optimization method for deep learning applications. While practitioners prefer Adam because it requires less parameter tuning, its use is problemat…
Better Parameter-free Stochastic Optimization with ODE Updates for Coin-Betting
Keyi Chen, John Langford, Francesco Orabona
Parameter-free stochastic gradient descent (PFSGD) algorithms do not require setting learning rates while achieving optimal theoretical performance. In practical applications, howe…
Temporal Variability in Implicit Online Learning
Nicolò Campolongo, Francesco Orabona
In the setting of online learning, Implicit algorithms turn out to be highly successful from a practical standpoint. However, the tightest regret analyses only show marginal improv…
Online Learning: A Modern Introduction Using Convex Optimization
Francesco Orabona
In this book, I introduce the concepts of online learning through a modern view based on convex optimization. Here, online learning refers to the framework of regret minimization u…
Parameter-Free Locally Differentially Private Stochastic Subgradient Descent
Kwang-Sung Jun, Francesco Orabona
We consider the problem of minimizing a convex risk with stochastic subgradients guaranteeing -locally differentially private (-LDP). While it has been shown that stochastic…
Kernel Truncated Randomized Ridge Regression: Optimal Rates and Low Noise Acceleration
Kwang-Sung Jun, Ashok Cutkosky, Francesco Orabona
In this paper, we consider the nonparametric least square regression in a Reproducing Kernel Hilbert Space (RKHS). We propose a new randomized algorithm that has optimal generaliza…
Momentum-Based Variance Reduction in Non-Convex SGD
Ashok Cutkosky, Francesco Orabona
Variance reduction has emerged in recent years as a strong competitor to stochastic gradient descent in non-convex problems, providing the first algorithms to improve upon the conv…
Parameter-Free Online Convex Optimization with Sub-Exponential Noise
Kwang-Sung Jun, Francesco Orabona
We consider the problem of unconstrained online convex optimization (OCO) with sub-exponential noise, a strictly more general problem than the standard OCO. In this setting, the le…
Surrogate Losses for Online Learning of Stepsizes in Stochastic Non-Convex Optimization
Zhenxun Zhuang, Ashok Cutkosky, Francesco Orabona
Stochastic Gradient Descent (SGD) has played a central role in machine learning. However, it requires a carefully hand-picked stepsize for fast convergence, which is notoriously te…
Black-Box Reductions for Parameter-free Online Learning in Banach Spaces
Ashok Cutkosky, Francesco Orabona
We introduce several new black-box reductions that significantly improve the design of adaptive and parameter-free online learning algorithms by simplifying analysis, improving reg…
Training Deep Networks without Learning Rates Through Coin Betting
Francesco Orabona, Tatiana Tommasi
Deep learning methods achieve state-of-the-art performance in many application scenarios. Yet, these methods require a significant amount of hyperparameters tuning in order to achi…
Efficient Online Bandit Multiclass Learning with Regret
Alina Beygelzimer, Francesco Orabona, Chicheng Zhang
We present an efficient second-order algorithm with regret for the bandit online multiclass problem. The regret bound holds simultaneously with respe…
Coin Betting and Parameter-Free Online Learning
Francesco Orabona, Dávid Pál
In the recent years, a number of parameter-free algorithms have been developed for online linear optimization over Hilbert spaces and for learning with expert advice. These algorit…