Integrated Model, Batch and Domain Parallelism in Training Neural Networks

Amir Gholami, Ariful Azad, Peter Jin +2

We propose a new integrated method of exploiting model, batch and domain parallelism for the training of deep neural networks (DNNs) on large distributed-memory computers using min…