Gradient Noise Convolution (GNC): Smoothing Loss Function for Distributed Large-Batch SGD

Kosuke Haruki, Taiji Suzuki, Yohei Hamakawa +4

Large-batch stochastic gradient descent (SGD) is widely used for training in distributed deep learning because of its training-time efficiency, however, extremely large-batch SGD l…