6 papers
Stochastic Saddle Avoidance Beyond Unit Excitation and Smoothness: A Pathwise Lyapunov-Perron Framework
Junwen Qiu, Bohao Ma, Andre Milzarek +1
Unit excitation (UE) is a common assumption in stochastic saddle avoidance: the stochastic error must have a uniformly positive component along every direction, in expectation. Thi…
Random Reshuffling with Momentum: Complexity Bounds and Last-iterate Convergence
Junwen Qiu, Bohao Ma, Andre Milzarek
Random reshuffling with momentum (RRM) corresponds to the SGD optimizer with the 'momentum' option enabled, as found in many machine learning libraries such as PyTorch and TensorFl…
Schattor: Schatten-family methods for deep learning optimization
Bohao Ma, Junyu Zhang, Chuan He
Modern deep learning optimization features heterogeneous parameter structures, noisy gradients, and highly nonconvex landscapes, posing significant challenges for both algorithm de…
A Single-Loop Regularized Newton Method for Nonconvex-Strongly-Concave Minimax Optimization
Bohao Ma, Nachuan Xiao, Junyu Zhang
For smooth nonconvex-strongly-concave minimax problems, existing second-order methods share a common double-loop structure where the inner maximization is solved to sufficiently hi…
Line-search and Adaptive Step Sizes for Nonconvex-strongly-concave Minimax Optimization
Bohao Ma, Nachuan Xiao, Junyu Zhang
In this paper, we propose a novel reformulation of the smooth nonconvex-strongly-concave (NC-SC) minimax problems that casts the problem as a joint minimization. We show that our r…
Convergence of SGD with momentum in the nonconvex case: A time window-based analysis
Junwen Qiu, Bohao Ma, Andre Milzarek
The stochastic gradient descent method with momentum (SGDM) is a common approach for solving large-scale and stochastic optimization problems. Despite its popularity, the convergen…