On Scale-out Deep Learning Training for Cloud and HPC

Abstract

The exponential growth in use of large deep neural networks has accelerated the need for training these deep neural networks in hours or even minutes. This can only be achieved through scalable and efficient distributed training, since a single node/card cannot satisfy the compute, memory, and I/O requirements of today's state-of-the-art deep neural networks. However, scaling synchronous Stochastic Gradient Descent (SGD) is still a challenging problem and requires continued research/development. This entails innovations spanning algorithms, frameworks, communication libraries, and system design. In this paper, we describe the philosophy, design, and implementation of Intel Machine Learning Scalability Library (MLSL) and present proof-points demonstrating scaling DL training on 100s to 1000s of nodes across Cloud and HPC systems.

Accepted in SysML 2018 conference

References in corpus (6)

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Don't Decay the Learning Rate, Increase the Batch Size
• On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima
• Distributed Deep Learning Using Synchronous Stochastic Gradient Descent
• 8-Bit Approximations for Parallelism in Deep Learning
• Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to train

Cited by in corpus (4)

• Scalable Distributed DNN Training using TensorFlow and CUDA-Aware MPI: Characterization, Designs, and Performance Evaluation
• HPC AI500: The Methodology, Tools, Roofline Performance Models, and Metrics for Benchmarking HPC AI Systems
• High-Performance Deep Learning via a Single Building Block
• Beyond the Memory Wall: A Case for Memory-centric HPC System for Deep Learning