Accelerating Distributed K-FAC with Smart Parallelism of Computing and Communication Tasks

Shaohuai Shi, Lin Zhang, Bo Li

Distributed training with synchronous stochastic gradient descent (SGD) on GPU clusters has been widely used to accelerate the training process of deep models. However, SGD only ut…