Reducing Data Motion to Accelerate the Training of Deep Neural Networks

Abstract

This paper reduces the cost of DNNs training by decreasing the amount of data movement across heterogeneous architectures composed of several GPUs and multicore CPU devices. In particular, this paper proposes an algorithm to dynamically adapt the data representation format of network weights during training. This algorithm drives a compression procedure that reduces data size before sending them over the parallel system. We run an extensive evaluation campaign considering several up-to-date deep neural network models and two high-end parallel architectures composed of multiple GPUs and CPU multicore chips. Our solution achieves average performance improvements from 6.18\% up to 11.91\%.

References in corpus (7)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• One weird trick for parallelizing convolutional neural networks
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Building high-level features using large scale unsupervised learning
• Scaling Deep Learning on GPU and Knights Landing clusters
• A Study of BFLOAT16 for Deep Learning Training