PowerGossip: Practical Low-Rank Communication Compression in Decentralized Deep Learning

Thijs Vogels, Sai Praneeth Karimireddy, Martin Jaggi

Lossy gradient compression has become a practical tool to overcome the communication bottleneck in centrally coordinated distributed training of machine learning models. However, a…