Quantized Distributed Training of Large Models with Convergence Guarantees

Ilia Markov, Adrian Vladu, Qi Guo +1

Communication-reduction techniques are a popular way to improve scalability in data-parallel training of deep neural networks (DNNs). The recent emergence of large language models…