54 citations · 225 across the 30 of their papers we have counts for
11 papers · 1 filter
Ori: Drift-Aware Streaming -PCA for Very Low-Rank LLM Training
Xinyu Luo, Jincheng Zhou, Mengbo Wang +5
A prominent family of low-rank LLM optimizers, pioneered by GaLore, performs gradient-subspace projection: at each step, they project the gradient into a low-dimensional subspace a…
MuonIO: Principled Norm-Aware Descent for Embedding Tables and Language Model Heads
Linkai Ma, Xinyu Luo, Mengbo Wang +3
The Muon optimizer derives its update rule for hidden linear layers by solving a local linearization of the loss penalized by the spectral norm, motivated by an RMS-stability argum…
Near-Optimal Acceleration for Smooth / Nondual Convex First-Order Oracle Optimization
David Martínez-Rubio, Brian Bullins, Cristóbal Guzmán +1
We study the optimization of convex objectives with -Hölder-continuous gradients in over , . (MG26) provides selectors with a movement bound fo…
The First-Order Oracle Complexity of Lipschitz Convex Optimization in Nondual Settings
David Martínez-Rubio, Brian Bullins, Cristóbal Guzmán +1
We study first-order black-box convex optimization over an -ball for objectives Lipschitz in the -norm, solving in the affirmative the nonsmooth version of the COLT…
Spectral Saliency for Machine Unlearning
Cedar Site Bai, Amber Yijia Zheng, Raymond A. Yeh +1
Machine unlearning (MU) aims to remove the influence of specific training data while preserving model utility. As the name suggests, MU can be viewed as the inverse of learning, us…
On the Limits of Biased Derivative Information for Nonconvex Stochastic Optimization
Anant Shyam, Brian Bullins
We consider the problem of finding -stationary points for , i.e., such that , for smooth, non-convex objectives, where the deri…
Can Entry-Wise Clipping Give Spectral Control of Stochastic Gradients?
Zitao Song, Cedar Site Bai, Zhe Zhang +2
Training instabilities such as loss spikes are frequently the result of stochastic gradient noise. Because of rare expressions in language training data, and multiple layer composi…
Mirror-Free Proximal Methods
Abhijeet Vyas, Brian Bullins
We present a \emph{mirror-free} mirror prox (MFMP) algorithm, which extends the classic approach of Nemirovski (2004) to allow for proximal-like updates without the explicit need f…
Beyond First-Order Methods for -Structured Non-Monotone Variational Inequalities
Abhijeet Vyas, Brian Bullins
We propose novel high-order algorithms for a class of -structured non-monotone variational inequalities. In particular, work by Diakonikolas et al. (2021), which introduced…
Online Min-Max Optimization: From Individual Regrets to Cumulative Saddle Points
Abhijeet Vyas, Brian Bullins
We propose and study an online version of min-max optimization based on cumulative saddle points under a variety of performance measures beyond convex-concave settings. After first…
Decoupling Variance and Scale-Invariant Updates in Adaptive Gradient Descent for Unified Vector and Matrix Optimization
Zitao Song, Cedar Site Bai, Zhe Zhang +2
Adaptive methods like Adam have become the standard for large-scale vector and Euclidean optimization due to their coordinate-wise adaptation with a second-orde…