8 papers
Solving bilevel optimization via sequential minimax optimization
Zhaosong Lu, Sanyou Mei
In this paper we propose a sequential minimax optimization (SMO) method for solving a class of constrained bilevel optimization problems in which the lower-level part is a possibly…
Accelerated stochastic first-order method for convex optimization under heavy-tailed noise
Chuan He, Zhaosong Lu, Bowen Li
We study convex composite optimization problems, where the objective function is given by the sum of a prox-friendly function and a convex function whose subgradients are estimated…
Low-rank Orthogonalization for Large-scale Matrix Optimization with Applications to Foundation Model Training
Chuan He, Zhanwang Deng, Zhaosong Lu
Neural network (NN) training is inherently a large-scale matrix optimization problem, yet the matrix structure of NN parameters has long been overlooked. Recently, the optimizer Mu…
First-order methods for stochastic and finite-sum convex optimization with deterministic constraints
Zhaosong Lu, Yifeng Xiao
In this paper, we study a class of stochastic and finite-sum convex optimization problems with deterministic constraints. Existing methods typically aim to find an -$expectedly\…
Complexity of normalized stochastic first-order methods with momentum under heavy-tailed noise
Chuan He, Zhaosong Lu, Defeng Sun +1
In this paper, we propose practical normalized stochastic first-order methods with Polyak momentum, multi-extrapolated momentum, and recursive momentum for solving unconstrained op…
Variance-reduced first-order methods for deterministically constrained stochastic nonconvex optimization with strong convergence guarantees
Zhaosong Lu, Sanyou Mei, Yifeng Xiao
In this paper, we study a class of deterministically constrained stochastic optimization problems. Existing methods typically aim to find an -stochastic stationary point, where…
Newton-CG methods for nonconvex unconstrained optimization with Hölder continuous Hessian
Chuan He, Heng Huang, Zhaosong Lu
In this paper we consider a nonconvex unconstrained optimization problem minimizing a twice differentiable objective function with Hölder continuous Hessian. Specifically, we first…
Primal-dual extrapolation methods for monotone inclusions under local Lipschitz continuity
Zhaosong Lu, Sanyou Mei
In this paper we consider a class of monotone inclusion (MI) problems of finding a zero of the sum of two monotone operators, in which one operator is maximal monotone while the ot…