Distributed Training of Deep Neural Networks: Theoretical and Practical Limits of Parallel Scalability

Janis Keuper, Franz-Josef Pfreundt

This paper presents a theoretical analysis and practical evaluation of the main bottlenecks towards a scalable distributed solution for the training of Deep Neuronal Networks (DNNs…