Accelerate Model Parallel Training by Using Efficient Graph Traversal Order in Device Placement

Tianze Wang, Amir H. Payberah, Desta Haileselassie Hagos +1

Modern neural networks require long training to reach decent performance on massive datasets. One common approach to speed up training is model parallelization, where large neural…