Flexible Communication for Optimal Distributed Learning over Unpredictable Networks

Sahil Tyagi, Martin Swany

Gradient compression alleviates expensive communication in distributed deep learning by sending fewer values and its corresponding indices, typically via Allgather (AG). Training w…