Communication-Efficient, 2D Parallel Stochastic Gradient Descent for Distributed-Memory Optimization

Aditya Devarakonda, Ramakrishnan Kannan

Distributed-memory implementations of numerical optimization algorithm, such as stochastic gradient descent (SGD), require interprocessor communication at every iteration of the al…