CROSSBOW: Scaling Deep Learning with Small Batch Sizes on Multi-GPU Servers

Abstract

Deep learning models are trained on servers with many GPUs, and training must scale with the number of GPUs. Systems such as TensorFlow and Caffe2 train models with parallel synchronous stochastic gradient descent: they process a batch of training data at a time, partitioned across GPUs, and average the resulting partial gradients to obtain an updated global model. To fully utilise all GPUs, systems must increase the batch size, which hinders statistical efficiency. Users tune hyper-parameters such as the learning rate to compensate for this, which is complex and model-specific. We describe CROSSBOW, a new single-server multi-GPU system for training deep learning models that enables users to freely choose their preferred batch size - however small - while scaling to multiple GPUs. CROSSBOW uses many parallel model replicas and avoids reduced statistical efficiency through a new synchronous training method. We introduce SMA, a synchronous variant of model averaging in which replicas independently explore the solution space with gradient descent, but adjust their search synchronously based on the trajectory of a globally-consistent average model. CROSSBOW achieves high hardware efficiency with small batch sizes by potentially training multiple model replicas per GPU, automatically tuning the number of replicas to maximise throughput. Our experiments show that CROSSBOW improves the training time of deep learning models on an 8-GPU server by 1.3-4x compared to TensorFlow.

References in corpus (17)

• Deep Residual Learning for Image Recognition
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• One weird trick for parallelizing convolutional neural networks
• The Loss Surfaces of Multilayer Networks
• Revisiting Distributed Synchronous SGD
• Horovod: fast and easy distributed deep learning in TensorFlow
• Deep Voice: Real-time Neural Text-to-Speech
• Revisiting Small Batch Training for Deep Neural Networks
• The Microsoft 2016 Conversational Speech Recognition System
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• Three Factors Influencing Minima in SGD
• Measuring the Effects of Data Parallelism on Neural Network Training
• Towards Optimal One Pass Large Scale Learning with Averaged Stochastic Gradient Descent
• YellowFin and the Art of Momentum Tuning

Cited by in corpus (5)

• The Evolution of Distributed Systems for Graph Neural Networks and their Origin in Graph Processing and Deep Learning: A Survey
• Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools
• Fast and Flexible Human Pose Estimation with HyperPose
• Deep-Edge: An Efficient Framework for Deep Learning Model Update on Heterogeneous Edge
• Optimizing Multi-GPU Parallelization Strategies for Deep Learning Training