3 citations · 6 across the 11 of their papers we have counts for
11 papers
FlashBoB: I/O-Efficient Exact Backward-over-Backward for Softmax Attention
Anthony Givans, Michael Crawshaw, Mingrui Liu
Transformer models built on the attention mechanism have become a central building block in modern deep learning, yet softmax attention remains a major bottleneck for long-context…
How Edge of Stability Hinders SCAFFOLD in Federated Optimization
Anant Khandelwal, Michael Crawshaw, Mingrui Liu
In federated learning, it is well known that heterogeneous data can (in theory) slow down optimization, and much effort has been directed at designing optimization algorithms that…
CURE-OOD: Benchmarking Out-of-Distribution Detection for Survival Prediction
Wenjie Zhao, Jia Li, Mingrui Liu +2
``How long can I live and remain free of cancer?'' is often the first question a patient asks after receiving a cancer diagnosis and treatment. Accurate survival prediction helps a…
Tight Bounds for Logistic Regression with Large Stepsize Gradient Descent in Low Dimension
Michael Crawshaw, Mingrui Liu
We consider the optimization problem of minimizing the logistic loss with gradient descent to train a linear model for binary classification with separable data. With a budget of $…
An Exploration of Non-Euclidean Gradient Descent: Muon and its Many Variants
Michael Crawshaw, Chirag Modi, Mingrui Liu +1
To define a steepest descent method over a neural network, we need to choose a norm for each layer, a way to aggregate these norms across layers, and whether to use normalization.…
Constant Stepsize Local GD for Logistic Regression: Acceleration by Instability
Michael Crawshaw, Blake Woodworth, Mingrui Liu
Existing analysis of Local (Stochastic) Gradient Descent for heterogeneous objectives requires stepsizes where is the communication interval, which ensures monotoni…
Complexity Lower Bounds of Adaptive Gradient Algorithms for Non-convex Stochastic Optimization under Relaxed Smoothness
Michael Crawshaw, Mingrui Liu
Recent results in non-convex stochastic optimization demonstrate the convergence of popular adaptive algorithms (e.g., AdaGrad) under the -smoothness condition, but the…
Local Steps Speed Up Local GD for Heterogeneous Distributed Logistic Regression
Michael Crawshaw, Blake Woodworth, Mingrui Liu
We analyze two variants of Local Gradient Descent applied to distributed logistic regression with heterogeneous, separable data and show convergence at the rate for l…
Federated Learning under Periodic Client Participation and Heterogeneous Data: A New Communication-Efficient Algorithm and Analysis
Michael Crawshaw, Mingrui Liu
In federated learning, it is common to assume that clients are always available to participate in training, which may not be feasible with user devices in practice. Recent works an…
EPISODE: Episodic Gradient Clipping with Periodic Resampled Corrections for Federated Learning with Heterogeneous Data
Michael Crawshaw, Yajie Bao, Mingrui Liu
Gradient clipping is an important technique for deep neural networks with exploding gradients, such as recurrent neural networks. Recent studies have shown that the loss functions…
Fast Composite Optimization and Statistical Recovery in Federated Learning
Yajie Bao, Michael Crawshaw, Shan Luo +1
As a prevalent distributed learning paradigm, Federated Learning (FL) trains a global model on a massive amount of devices with infrequent communication. This paper investigates a…