Do We Need Asynchronous SGD? On the Near-Optimality of Synchronous Methods

Grigory Begunov, Alexander Tyurin

Modern distributed optimization methods mostly rely on traditional synchronous approaches, despite substantial recent progress in asynchronous optimization. We revisit Synchronous…