Distributed Training Large-Scale Deep Architectures

Abstract

Scale of data and scale of computation infrastructures together enable the current deep learning renaissance. However, training large-scale deep architectures demands both algorithmic improvement and careful system configuration. In this paper, we focus on employing the system approach to speed up large-scale training. Via lessons learned from our routine benchmarking effort, we first identify bottlenecks and overheads that hinter data parallelism. We then devise guidelines that help practitioners to configure an effective system and fine-tune parameters to achieve desired speedup. Specifically, we develop a procedure for setting minibatch size and choosing computation algorithms. We also derive lemmas for determining the quantity of key components such as the number of GPUs and parameter servers. Experiments and examples show that these guidelines help effectively speed up large-scale deep learning training.

References in corpus (16)

• Caffe: Convolutional Architecture for Fast Feature Embedding
• Recurrent Neural Network Regularization
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• cuDNN: Efficient Primitives for Deep Learning
• One weird trick for parallelizing convolutional neural networks
• Revisiting Distributed Synchronous SGD
• Fast Training of Convolutional Networks through FFTs
• Practical recommendations for gradient-based training of deep architectures
• Fast Convolutional Nets With fbfft: A GPU Performance Evaluation
• Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
• Comparative Study of Deep Learning Software Frameworks
• Benchmarking State-of-the-Art Deep Learning Software Tools
• Poseidon: A System Architecture for Efficient GPU-based Deep Learning on Multiple Machines
• Caffe con Troll: Shallow Ideas to Speed Up Deep Learning
• Speech Recognition with Deep Recurrent Neural Networks

Cited by in corpus (2)

• Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs
• Hoard: A Distributed Data Caching System to Accelerate Deep Learning Training on the Cloud