Omnivore: An Optimizer for Multi-device Deep Learning on CPUs and GPUs

Abstract

We study the factors affecting training time in multi-device deep learning systems. Given a specification of a convolutional neural network, our goal is to minimize the time to train this model on a cluster of commodity CPUs and GPUs. We first focus on the single-node setting and show that by using standard batching and data-parallel techniques, throughput can be improved by at least 5.5x over state-of-the-art systems on CPUs. This ensures an end-to-end training speed directly proportional to the throughput of a device regardless of its underlying hardware, allowing each node in the cluster to be treated as a black box. Our second contribution is a theoretical and empirical study of the tradeoffs affecting end-to-end training time in a multiple-device setting. We identify the degree of asynchronous parallelization as a key factor affecting both hardware and statistical efficiency. We see that asynchrony can be viewed as introducing a momentum term. Our results imply that tuning momentum is critical in asynchronous parallel configurations, and suggest that published results that have not been fully tuned might report suboptimal performance for some configurations. For our third contribution, we use our novel understanding of the interaction between system and optimization dynamics to provide an efficient hyperparameter optimizer. Our optimizer involves a predictive model for the total time to convergence and selects an allocation of resources to minimize that time. We demonstrate that the most popular distributed deep learning systems fall within our tradeoff space, but do not optimize within the space. By doing this optimization, our prototype runs 1.9x to 12x faster than the fastest state-of-the-art systems.

References in corpus (9)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• TensorFlow: A system for large-scale machine learning
• Practical Bayesian Optimization of Machine Learning Algorithms
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Revisiting Distributed Synchronous SGD
• Text Understanding from Scratch
• Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization
• SparkNet: Training Deep Networks in Spark

Cited by in corpus (18)

• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• Database Meets Deep Learning: Challenges and Opportunities
• BlinkML: Efficient Maximum Likelihood Estimation with Probabilistic Guarantees
• Hemingway: Modeling Distributed Optimization Algorithms
• Deep Learning at 15PF: Supervised and Semi-Supervised Classification for Scientific Data
• Infrastructure for Usable Machine Learning: The Stanford DAWN Project
• Distributed Learning of Deep Neural Networks using Independent Subnet Training
• Scheduling Computation Graphs of Deep Learning Models on Manycore CPUs
• Stochastic Gradient Descent on Highly-Parallel Architectures
• PANDA: Facilitating Usable AI Development
• Accumulated Gradient Normalization
• Dynamic Stale Synchronous Parallel Distributed Training for Deep Learning
• Distributed Machine Learning through Heterogeneous Edge Systems
• Accelerating Asynchronous Stochastic Gradient Descent for Neural Machine Translation
• Heterogeneity-aware Gradient Coding for Straggler Tolerance
• Adaptive Elastic Training for Sparse Deep Learning on Heterogeneous Multi-GPU Servers
• Heterogeneous CPU+GPU Stochastic Gradient Descent Algorithms
• Machine Learning Systems for Intelligent Services in the IoT: A Survey