8 papers
Low-rank Orthogonalization for Large-scale Matrix Optimization with Applications to Foundation Model Training
Chuan He, Zhanwang Deng, Zhaosong Lu
Neural network (NN) training is inherently a large-scale matrix optimization problem, yet the matrix structure of NN parameters has long been overlooked. Recently, the optimizer Mu…
Complexity of normalized stochastic first-order methods with momentum under heavy-tailed noise
Chuan He, Zhaosong Lu, Defeng Sun +1
In this paper, we propose practical normalized stochastic first-order methods with Polyak momentum, multi-extrapolated momentum, and recursive momentum for solving unconstrained op…
Solving bilevel optimization via sequential minimax optimization
Zhaosong Lu, Sanyou Mei
In this paper we propose a sequential minimax optimization (SMO) method for solving a class of constrained bilevel optimization problems in which the lower-level part is a possibly…
Accelerated stochastic first-order method for convex optimization under heavy-tailed noise
Chuan He, Zhaosong Lu
We study convex composite optimization problems, where the objective function is given by the sum of a prox-friendly function and a convex function whose subgradients are estimated…
Variance-reduced first-order methods for deterministically constrained stochastic nonconvex optimization with strong convergence guarantees
Zhaosong Lu, Sanyou Mei, Yifeng Xiao
In this paper, we study a class of deterministically constrained stochastic optimization problems. Existing methods typically aim to find an -stochastic stationary point, where…
First-order methods for stochastic and finite-sum convex optimization with deterministic constraints
Zhaosong Lu, Yifeng Xiao
In this paper, we study a class of stochastic and finite-sum convex optimization problems with deterministic constraints. Existing methods typically aim to find an -$expectedly…