4 papers
Decoupled DiLoCo for Resilient Distributed Pre-training
Arthur Douillard, Keith Rush, Yani Donchev +14
Modern large-scale language model pre-training relies heavily on the single program multiple data (SPMD) paradigm, which requires tight coupling across accelerators. Due to this co…
Constant Stepsize Local GD for Logistic Regression: Acceleration by Instability
Michael Crawshaw, Blake Woodworth, Mingrui Liu
Existing analysis of Local (Stochastic) Gradient Descent for heterogeneous objectives requires stepsizes where is the communication interval, which ensures monoton…
Local Steps Speed Up Local GD for Heterogeneous Distributed Logistic Regression
Michael Crawshaw, Blake Woodworth, Mingrui Liu
We analyze two variants of Local Gradient Descent applied to distributed logistic regression with heterogeneous, separable data and show convergence at the rate for l…
Gradient Descent Converges Linearly to Flatter Minima than Gradient Flow in Shallow Linear Networks
Pierfrancesco Beneventano, Blake Woodworth
We study the gradient descent (GD) dynamics of a depth-2 linear neural network with a single input and output. We show that GD converges at an explicit linear rate to a global mini…