4 papers · 1 filter
Monarch-Muon: Structure-Aware Muon for Fast and Memory-Efficient Pretraining
Alexander Miasnikov, Ruslan Kabirov, Gleb Molodtsov +7
Training large language models is constrained by both wall-clock time and memory footprint, and recent matrix-aware optimizers sharpen this tradeoff: Muon can improve optimization…
Lower Bounds For Gradient-Free Convex Optimization And Convex-Concave Saddle-Point Problems
Yuriy Dorn, Darina Dvinskikh, Timofei Loginov +4
We study the query complexity of optimization with exact scalar-value information. For globally -smooth convex functions on with a minimizer in a Euclidean ball of…
The Complexity of Convex Optimization with Mismatched Geometry
Timofei Loginov, Alexander Gasnikov, Yuriy Dorn +4
Optimal first-order methods on non-Euclidean domains such as the ball pair the prox-function with the norm in which smoothness…
Why SGD is not Brownian Motion: A New Perspective on Stochastic Dynamics
Igor Ignashin, Anna Radovskaya, Andrew Semenov +7
Stochastic Gradient Descent (SGD) is commonly modeled as a Langevin process, assuming that minibatch noise acts as Brownian motion. However, this approximation relies on a continuo…