Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools

Abstract

Deep Learning (DL) has had an immense success in the recent past, leading to state-of-the-art results in various domains such as image recognition and natural language processing. One of the reasons for this success is the increasing size of DL models and the proliferation of vast amounts of training data being available. To keep on improving the performance of DL, increasing the scalability of DL systems is necessary. In this survey, we perform a broad and thorough investigation on challenges, techniques and tools for scalable DL on distributed infrastructures. This incorporates infrastructures for DL, methods for parallel DL training, multi-tenant resource scheduling and the management of training and model data. Further, we analyze and compare 11 current open-source DL frameworks and tools and investigate which of the techniques are commonly implemented in practice. Finally, we highlight future research trends in DL systems that deserve further research.

arXiv commentaccepted at ACM Computing Surveys, to appear

References in corpus (46)

• Scikit-learn: Machine Learning in Python
• Deep Learning in Neural Networks: An Overview
• A Survey on Deep Learning in Medical Image Analysis
• Deep Learning with Differential Privacy
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• An overview of gradient descent optimization algorithms
• A Brief Survey of Deep Reinforcement Learning
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Neural Architecture Search with Reinforcement Learning
• Federated Learning: Strategies for Improving Communication Efficiency
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Recent Trends in Deep Learning Based Natural Language Processing
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Deep Big Simple Neural Nets Excel on Handwritten Digit Recognition
• cuDNN: Efficient Primitives for Deep Learning
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• One weird trick for parallelizing convolutional neural networks
• QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
• Neuromorphic computing with multi-memristive synapses
• TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning
• Horovod: fast and easy distributed deep learning in TensorFlow
• Designing Neural Network Architectures using Reinforcement Learning
• Deep Learning Scaling is Predictable, Empirically
• Revisiting Small Batch Training for Deep Neural Networks
• Rosetta: Large scale system for text detection and recognition in images
• A generic framework for privacy preserving deep learning
• Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect
• A Reliable Effective Terascale Linear Learning System
• GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
• Device Placement Optimization with Reinforcement Learning
• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• Robust Loss Functions under Label Noise for Deep Neural Networks
• Database Meets Deep Learning: Challenges and Opportunities
• PipeDream: Fast and Efficient Pipeline Parallel DNN Training
• High-Performance Distributed ML at Scale through Parameter Server Consistency Models
• Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads
• High-Accuracy Low-Precision Training
• ChainerMN: Scalable Distributed Deep Learning Framework
• Distributed TensorFlow with MPI
• Hoard: A Distributed Data Caching System to Accelerate Deep Learning Training on the Cloud
• CROSSBOW: Scaling Deep Learning with Small Batch Sizes on Multi-GPU Servers
• The streaming rollout of deep networks - towards fully model-parallel execution
• Improving the Expressiveness of Deep Learning Frameworks with Recursion
• Accelerated Training for CNN Distributed Deep Learning through Automatic Resource-Aware Layer Placement

Cited by in corpus (8)

• Database Meets Deep Learning: Challenges and Opportunities
• Communication optimization strategies for distributed deep neural network training: A survey
• Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems
• Video Big Data Analytics in the Cloud: A Reference Architecture, Survey, Opportunities, and Open Research Issues
• Communication-Efficient Edge AI: Algorithms and Systems
• LayerPipe: Accelerating Deep Neural Network Training by Intra-Layer and Inter-Layer Gradient Pipelining and Multiprocessor Scheduling
• Scavenger: A Cloud Service for Optimizing Cost and Performance of ML Training
• Machine Learning Systems for Intelligent Services in the IoT: A Survey