Partial Parameter Updates for Efficient Distributed Training

Anastasiia Filippova, Angelos Katharopoulos, David Grangier +1

We introduce a memory- and compute-efficient method for low-communication distributed training. Existing methods reduce communication by performing multiple local updates between i…