5 papers
Communication-Efficient Distributed Training for Collaborative Flat Optima Recovery in Deep Learning
Tolga Dimlioglu, Anna Choromanska
We study centralized distributed data parallel training of deep neural networks (DNNs), aiming to improve the trade-off between communication efficiency and model performance of th…
Outer-Momentum Restarting in High-Dimensional Two-Phase Optimization
Kristi Topollai, Allan Ma, Tolga Dimlioglu +2
Communication-efficient distributed optimizers such as DiLoCo reduce synchronization costs by letting workers perform many local updates before aggregating their progress with an o…
Worker Disagreement Reveals Sharp Directions in Local SGD
Tolga Dimlioglu, Kristi Topollai, Anna Choromanska
Deep neural network training often exhibits highly anisotropic loss geometry, where a few sharp dominant Hessian directions coexist with a large flatter bulk. Gradients tend to ali…
Understanding Quantization of Optimizer States in LLM Pre-training: Dynamics of State Staleness and Effectiveness of State Resets
Kristi Topollai, Anna Choromanska
Quantizing optimizer states is becoming an important ingredient of memory-efficient large-scale pre-training, but the resulting optimizer dynamics remain only partially understood.…
A Survey of Optimization Methods for Training DL Models: Theoretical Perspective on Convergence and Generalization
Jing Wang, Anna Choromanska
As data sets grow in size and complexity, it is becoming more difficult to pull useful features from them using hand-crafted feature extractors. For this reason, deep learning (DL)…