407 citations · 420 across the 15 of their papers we have counts for
15 papers
FlashBoB: I/O-Efficient Exact Backward-over-Backward for Softmax Attention
Anthony Givans, Michael Crawshaw, Mingrui Liu
Transformer models built on the attention mechanism have become a central building block in modern deep learning, yet softmax attention remains a major bottleneck for long-context…
How Edge of Stability Hinders SCAFFOLD in Federated Optimization
Anant Khandelwal, Michael Crawshaw, Mingrui Liu
In federated learning, it is well known that heterogeneous data can (in theory) slow down optimization, and much effort has been directed at designing optimization algorithms that…
Muon Does Not Converge on Convex Lipschitz Functions
Tetiana Parshakova, Ahmed Khaled, Michael Crawshaw +2
Muon and its variants have shown strong empirical performance in a variety of deep learning tasks. Existing convergence analyses of Muon rely on smoothness assumptions, though argu…
Non-Euclidean Gradient Descent Operates at the Edge of Stability
Rustem Islamov, Michael Crawshaw, Jeremy Cohen +1
The Edge of Stability (EoS) is a phenomenon where the sharpness (largest eigenvalue) of the Hessian approaches and then hovers near the stability threshold during gradient de…
Tight Bounds for Logistic Regression with Large Stepsize Gradient Descent in Low Dimension
Michael Crawshaw, Mingrui Liu
We consider the optimization problem of minimizing the logistic loss with gradient descent to train a linear model for binary classification with separable data. With a budget of $…
An Exploration of Non-Euclidean Gradient Descent: Muon and its Many Variants
Michael Crawshaw, Chirag Modi, Mingrui Liu +1
To define a steepest descent method over a neural network, we need to choose a norm for each layer, a way to aggregate these norms across layers, and whether to use normalization.…