Understanding Outer Optimizers in Local SGD: Learning Rates, Momentum, and Acceleration

Ahmed Khaled, Satyen Kale, Arthur Douillard +3

Modern machine learning often requires training with large batch size, distributed data, and massively parallel compute hardware (like mobile and other edge devices or distributed…