Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methods

Tim Tsz-Kit Lau, Weijian Li, Chenwei Xu +2

Modern deep neural networks often require distributed training with many workers due to their large size. As the number of workers increases, communication overheads become the mai…