CGX: Adaptive System Support for Communication-Efficient Deep Learning

Ilia Markov, Hamidreza Ramezanikebrya, Dan Alistarh

The ability to scale out training workloads has been one of the key performance enablers of deep learning. The main scaling approach is data-parallel GPU-based training, which has…