Democratizing Production-Scale Distributed Deep Learning
arXiv:1811.00143
Abstract
The interest and demand for training deep neural networks have been experiencing rapid growth, spanning a wide range of applications in both academia and industry. However, training them distributed and at scale remains difficult due to the complex ecosystem of tools and hardware involved. One consequence is that the responsibility of orchestrating these complex components is often left to one-off scripts and glue code customized for specific problems. To address these restrictions, we introduce \emph{Alchemist} - an internal service built at Apple from the ground up for \emph{easy}, \emph{fast}, and \emph{scalable} distributed training. We discuss its design, implementation, and examples of running different flavors of distributed training. We also present case studies of its internal adoption in the development of autonomous systems, where training times have been reduced by 10x to keep up with the ever-growing data collection.
References in corpus (11)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
- A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
- Tune: A Research Platform for Distributed Model Selection and Training
- Horovod: fast and easy distributed deep learning in TensorFlow
- Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
- Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes
- BigDL: A Distributed Deep Learning Framework for Big Data