37 citations · 60 across the 12 of their papers we have counts for
13 papers
Online Learning-guided Learning Rate Adaptation via Gradient Alignment
Ruichen Jiang, Ali Kavis, Aryan Mokhtari
The performance of an optimizer on large-scale deep learning models depends critically on fine-tuning the learning rate, often requiring an extensive grid search over base learning…
Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting
Sunny Sanyal, Hayden Prairie, Rudrajit Das +2
Fine-tuning a pre-trained model on a downstream task often degrades its original capabilities, a phenomenon known as "catastrophic forgetting". This is especially an issue when one…
Understanding Self-Supervised Learning via Gaussian Mixture Models
Parikshit Bansal, Ali Kavis, Sujay Sanghavi
Self-supervised learning attempts to learn representations from un-labeled data; it does so via a loss function that encourages the embedding of a point to be close to that of its…
Adaptive and Optimal Second-order Optimistic Methods for Minimax Optimization
Ruichen Jiang, Ali Kavis, Qiujiang Jin +2
We propose adaptive, line search-free second-order methods with optimal rate of convergence for solving convex-concave min-max problems. By means of an adaptive step size, our algo…
Universal Gradient Methods for Stochastic Convex Optimization
Anton Rodomanov, Ali Kavis, Yongtao Wu +2
We develop universal gradient methods for Stochastic Convex Optimization (SCO). Our algorithms automatically adapt not only to the oracle's noise but also to the Hölder smoothness…
Extra-Newton: A First Approach to Noise-Adaptive Accelerated Second-Order Methods
Kimon Antonakopoulos, Ali Kavis, Volkan Cevher
This work proposes a universal and adaptive second-order method for minimizing second-order smooth, convex functions. Our algorithm achieves convergence when the o…
Adaptive Stochastic Variance Reduction for Non-convex Finite-Sum Minimization
Ali Kavis, Stratis Skoulakis, Kimon Antonakopoulos +2
We propose an adaptive variance-reduction method, called AdaSpider, for minimization of -smooth, non-convex functions with a finite-sum structure. In essence, AdaSpider combines…
High Probability Bounds for a Class of Nonconvex Algorithms with AdaGrad Stepsize
Ali Kavis, Kfir Yehuda Levy, Volkan Cevher
In this paper, we propose a new, simplified high probability analysis of AdaGrad for smooth, non-convex problems. More specifically, we focus on a particular accelerated gradient (…
STORM+: Fully Adaptive SGD with Momentum for Nonconvex Optimization
Kfir Y. Levy, Ali Kavis, Volkan Cevher
In this work we investigate stochastic non-convex optimization problems where the objective is an expectation over smooth loss functions, and the goal is to find an approximate sta…
Double-Loop Unadjusted Langevin Algorithm
Paul Rolland, Armin Eftekhari, Ali Kavis +1
A well-known first-order method for sampling from log-concave probability distributions is the Unadjusted Langevin Algorithm (ULA). This work proposes a new annealing step-size sch…
On the Almost Sure Convergence of Stochastic Gradient Descent in Non-Convex Problems
Panayotis Mertikopoulos, Nadav Hallak, Ali Kavis +1
This paper analyzes the trajectories of stochastic gradient descent (SGD) to help understand the algorithm's convergence properties in non-convex problems. We first show that the s…
UniXGrad: A Universal, Adaptive Algorithm with Optimal Guarantees for Constrained Optimization
Ali Kavis, Kfir Y. Levy, Francis Bach +1
We propose a novel adaptive, accelerated algorithm for the stochastic constrained convex optimization setting. Our method, which is inspired by the Mirror-Prox method, \emph{simult…
Efficient learning of smooth probability functions from Bernoulli tests with guarantees
Paul Rolland, Ali Kavis, Alex Immer +2
We study the fundamental problem of learning an unknown, smooth probability function via pointwise Bernoulli tests. We provide a scalable algorithm for efficiently solving this pro…