310 citations · 496 across the 14 of their papers we have counts for
3 papers · 1 filter
Partial Parameter Updates for Efficient Distributed Training
Anastasiia Filippova, Angelos Katharopoulos, David Grangier +1
We introduce a memory- and compute-efficient method for low-communication distributed training. Existing methods reduce communication by performing multiple local updates between i…
Compute-Optimal Quantization-Aware Training
Aleksandr Dremov, David Grangier, Angelos Katharopoulos +1
Quantization-aware training (QAT) is a leading technique for improving the accuracy of quantized neural networks. Previous work has shown that decomposing training into a full-prec…
Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging
Pierre Ablin, Angelos Katharopoulos, Skyler Seto +1
Machine learning models are routinely trained on a mixture of different data domains. Different domain weights yield very different downstream performances. We propose the Soup-of-…