2.6k citations · 3.1k across the 36 of their papers we have counts for
13 papers · 1 filter
Near-Optimal Nonconvex-Strongly-Convex Bilevel Optimization with Fully First-Order Oracles
Lesi Chen, Yaohua Ma, Jingzhao Zhang
In this work, we consider bilevel optimization when the lower-level problem is strongly convex. Recent works show that with a Hessian-vector product (HVP) oracle, one can provably…
Online Control with Adversarial Disturbance for Continuous-time Linear Systems
Jingwei Li, Jing Dong, Can Chang +2
We study online control for continuous-time linear systems with finite sampling rates, where the objective is to design an online procedure that learns under non-stochastic noise a…
On Finding Small Hyper-Gradients in Bilevel Optimization: Hardness Results and Improved Analysis
Lesi Chen, Jing Xu, Jingzhao Zhang
Bilevel optimization reveals the inner structure of otherwise oblique optimization problems, such as hyperparameter tuning, neural architecture search, and meta-learning. A common…
Understanding the unstable convergence of gradient descent
Kwangjun Ahn, Jingzhao Zhang, Suvrit Sra
Most existing analyses of (stochastic) gradient descent rely on the condition that for -smooth costs, the step size is less than . However, many works have observed that in…
Sion's Minimax Theorem in Geodesic Metric Spaces and a Riemannian Extragradient Algorithm
Peiyuan Zhang, Jingzhao Zhang, Suvrit Sra
Deciding whether saddle points exist or are approximable for nonconvex-nonconcave problems is usually intractable. This paper takes a step towards understanding a broad class of no…
Beyond Worst-Case Analysis in Stochastic Approximation: Moment Estimation Improves Instance Complexity
Jingzhao Zhang, Hongzhou Lin, Subhro Das +2
We study oracle complexity of gradient based methods for stochastic approximation problems. Though in many settings optimal algorithms and tight lower bounds are known for such pro…
Complexity of Finding Stationary Points of Nonsmooth Nonconvex Functions
Jingzhao Zhang, Hongzhou Lin, Stefanie Jegelka +2
We provide the first non-asymptotic analysis for finding stationary points of nonsmooth, nonconvex functions. In particular, we study the class of Hadamard semi-differentiable func…
Why are Adaptive Methods Good for Attention Models?
Jingzhao Zhang, Sai Praneeth Karimireddy, Andreas Veit +4
While stochastic gradient descent (SGD) is still the \emph{de facto} algorithm in deep learning, adaptive methods like Clipped SGD/Adam have been observed to outperform SGD across…
Acceleration in First Order Quasi-strongly Convex Optimization by ODE Discretization
Jingzhao Zhang, Suvrit Sra, Ali Jadbabaie
We study gradient-based optimization methods obtained by direct Runge-Kutta discretization of the ordinary differential equation (ODE) describing the movement of a heavy-ball under…
Why gradient clipping accelerates training: A theoretical justification for adaptivity
Jingzhao Zhang, Tianxing He, Suvrit Sra +1
We provide a theoretical explanation for the effectiveness of gradient clipping in training deep neural networks. The key ingredient is a new smoothness condition derived from prac…
R-SPIDER: A Fast Riemannian Stochastic Optimization Algorithm with Curvature Independent Rate
Jingzhao Zhang, Hongyi Zhang, Suvrit Sra
We study smooth stochastic optimization problems on Riemannian manifolds. Via adapting the recently proposed SPIDER algorithm \citep{fang2018spider} (a variance reduced stochastic…
Achieving Acceleration in Distributed Optimization via Direct Discretization of the Heavy-Ball ODE
Jingzhao Zhang, César A. Uribe, Aryan Mokhtari +1
We develop a distributed algorithm for convex Empirical Risk Minimization, the problem of minimizing large but finite sum of convex functions over networks. The proposed algorithm…
Direct Runge-Kutta Discretization Achieves Acceleration
Jingzhao Zhang, Aryan Mokhtari, Suvrit Sra +1
We study gradient-based optimization methods obtained by directly discretizing a second-order ordinary differential equation (ODE) related to the continuous limit of Nesterov's acc…