54 citations · 225 across the 30 of their papers we have counts for
15 papers · 1 filter
Ori: Drift-Aware Streaming -PCA for Very Low-Rank LLM Training
Xinyu Luo, Jincheng Zhou, Mengbo Wang +5
A prominent family of low-rank LLM optimizers, pioneered by GaLore, performs gradient-subspace projection: at each step, they project the gradient into a low-dimensional subspace a…
MuonIO: Principled Norm-Aware Descent for Embedding Tables and Language Model Heads
Linkai Ma, Xinyu Luo, Mengbo Wang +3
The Muon optimizer derives its update rule for hidden linear layers by solving a local linearization of the loss penalized by the spectral norm, motivated by an RMS-stability argum…
Spectral Saliency for Machine Unlearning
Cedar Site Bai, Amber Yijia Zheng, Raymond A. Yeh +1
Machine unlearning (MU) aims to remove the influence of specific training data while preserving model utility. As the name suggests, MU can be viewed as the inverse of learning, us…
Can Entry-Wise Clipping Give Spectral Control of Stochastic Gradients?
Zitao Song, Cedar Site Bai, Zhe Zhang +2
Training instabilities such as loss spikes are frequently the result of stochastic gradient noise. Because of rare expressions in language training data, and multiple layer composi…
Online Min-Max Optimization: From Individual Regrets to Cumulative Saddle Points
Abhijeet Vyas, Brian Bullins
We propose and study an online version of min-max optimization based on cumulative saddle points under a variety of performance measures beyond convex-concave settings. After first…
Decoupling Variance and Scale-Invariant Updates in Adaptive Gradient Descent for Unified Vector and Matrix Optimization
Zitao Song, Cedar Site Bai, Zhe Zhang +2
Adaptive methods like Adam have become the standard for large-scale vector and Euclidean optimization due to their coordinate-wise adaptation with a second-orde…
Stacey: Promoting Stochastic Steepest Descent via Accelerated -Smooth Nonconvex Optimization
Xinyu Luo, Cedar Site Bai, Bolian Li +3
While popular optimization methods such as SGD, AdamW, and Lion depend on steepest descent updates in either or norms, there remains a critical gap in handli…
Model Immunization from a Condition Number Perspective
Amber Yijia Zheng, Cedar Site Bai, Brian Bullins +1
Model immunization aims to pre-train models that are difficult to fine-tune on harmful tasks while retaining their utility on other non-harmful tasks. Though prior work has shown e…
Federated Composite Saddle Point Optimization
Site Bai, Brian Bullins
Federated learning (FL) approaches for saddle point problems (SPP) have recently gained in popularity due to the critical role they play in machine learning (ML). Existing works mo…
Variance-Reduced Conservative Policy Iteration
Naman Agarwal, Brian Bullins, Karan Singh
We study the sample complexity of reducing reinforcement learning to a sequence of empirical risk minimization problems over the policy space. Such reductions-based algorithms exhi…
The Min-Max Complexity of Distributed Stochastic Convex Optimization with Intermittent Communication
Blake Woodworth, Brian Bullins, Ohad Shamir +1
We resolve the min-max complexity of distributed stochastic convex optimization (up to a log factor) in the intermittent communication setting, where machines work in parallel…
Is Local SGD Better than Minibatch SGD?
Blake Woodworth, Kumar Kshitij Patel, Sebastian U. Stich +5
We study local SGD (also known as parallel SGD and federated averaging), a natural and frequently used stochastic distributed optimization method. Its theoretical foundations are c…
Online Control with Adversarial Disturbances
Naman Agarwal, Brian Bullins, Elad Hazan +2
We study the control of a linear dynamical system with adversarial disturbances (as opposed to statistical noise). The objective we consider is one of regret: we desire an online c…
Efficient Full-Matrix Adaptive Regularization
Naman Agarwal, Brian Bullins, Xinyi Chen +4
Adaptive regularization methods pre-multiply a descent direction by a preconditioning matrix. Due to the large number of parameters of machine learning problems, full-matrix precon…
Not-So-Random Features
Brian Bullins, Cyril Zhang, Yi Zhang
We propose a principled method for kernel learning, which relies on a Fourier-analytic characterization of translation-invariant or rotation-invariant kernels. Our method produces…