2 papers
cs.LG2025
ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training
Adel Nabli, Louis Fournier, Pierre Erbacher +3
Training LLMs relies on distributed implementations using multiple GPUs to compute gradients in parallel with sharded optimizers. However, synchronizing gradients in data parallel…
cs.LG2024
WASH: Train your Ensemble with Communication-Efficient Weight Shuffling, then Average
Louis Fournier, Adel Nabli, Masih Aminbeidokhti +3
The performance of deep neural networks is enhanced by ensemble methods, which average the output of several models. However, this comes at an increased cost at inference. Weight a…