BAGUA: Scaling up Distributed Learning with System Relaxations

Shaoduo Gan, Xiangru Lian, Rui Wang +11

Recent years have witnessed a growing list of systems for distributed data-parallel training. Existing systems largely fit into two paradigms, i.e., parameter server and MPI-style…