Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
arXiv:1802.09941
Abstract
Deep Neural Networks (DNNs) are becoming an important tool in modern computing applications. Accelerating their training is a major challenge and techniques range from distributed algorithms to low-level circuit design. In this survey, we describe the problem from a theoretical perspective, followed by approaches for its parallelization. We present trends in DNN architectures and the resulting implications on parallelization strategies. We then review and model the different types of concurrency in DNNs: from the single operator, through parallelism in network inference and training, to distributed deep learning. We discuss asynchronous stochastic optimization, distributed system architectures, communication schemes, and neural architecture search. Based on those approaches, we extrapolate potential directions for parallelism in deep learning.
References in corpus (85)
- Deep Learning in Neural Networks: An Overview
- Distilling the Knowledge in a Neural Network
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
- Practical Bayesian Optimization of Machine Learning Algorithms
- Caffe: Convolutional Architecture for Fast Feature Embedding
- Neural Architecture Search with Reinforcement Learning
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
- End to End Learning for Self-Driving Cars
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
- Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
- BinaryConnect: Training Deep Neural Networks with binary weights during propagations
- DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Progressive Growing of GANs for Improved Quality, Stability, and Variation
- Do Deep Nets Really Need to be Deep?
- Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
- DARTS: Differentiable Architecture Search
- End-to-end Optimized Image Compression
- A guide to convolution arithmetic for deep learning
- HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
- Deep Learning with Limited Numerical Precision
- cuDNN: Efficient Primitives for Deep Learning
- Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification
- Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
- One weird trick for parallelizing convolutional neural networks
- Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
- QSGD: Communication-Efficient SGD via Gradient Quantization and Encoding
- Ternary Weight Networks
- Efficient Neural Architecture Search via Parameter Sharing
- Don't Decay the Learning Rate, Increase the Batch Size
- Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning
- Optimal Distributed Online Prediction using Mini-Batches
- Deep Reinforcement Learning: An Overview
- Training Deep Nets with Sublinear Memory Cost
- Large Batch Training of Convolutional Networks
- Train longer, generalize better: closing the generalization gap in large batch training of neural networks
- SMASH: One-Shot Model Architecture Search through HyperNetworks
- Xception: Deep Learning with Depthwise Separable Convolutions
- Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization
- XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
- One Model To Learn Them All
- Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions
- Why M Heads are Better than One: Training a Diverse Ensemble of Deep Networks
- Staleness-aware Async-SGD for Distributed Deep Learning
- Simple And Efficient Architecture Search for Convolutional Neural Networks
- A Linearly-Convergent Stochastic L-BFGS Algorithm
- Progressive Neural Architecture Search
- Deep Learning on FPGAs: Past, Present, and Future
- DeepArchitect: Automatically Designing and Training Deep Architectures
- TVM: An Automated End-to-End Optimizing Compiler for Deep Learning
- Parallel training of DNNs with Natural Gradient and Parameter Averaging
- Meta Learning Shared Hierarchies
- Deep Learning of Representations: Looking Forward
- Taming the Wild: A Unified Analysis of Hogwild!-Style Algorithms
- Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation
- 8-Bit Approximations for Parallelism in Deep Learning
- On the Origin of Deep Learning
- GossipGraD: Scalable Deep Learning using Gossip Communication based Asynchronous Gradient Descent
- Optimizing Performance of Recurrent Neural Networks on GPUs
- Scaling Deep Learning on GPU and Knights Landing clusters
- GPU Asynchronous Stochastic Gradient Descent to Speed Up Neural Network Training
- YellowFin and the Art of Momentum Tuning
- How to scale distributed deep learning?
- Memory-Efficient Backpropagation Through Time
- Poseidon: An Efficient Communication Architecture for Distributed Deep Learning on GPU Clusters
- Omnivore: An Optimizer for Multi-device Deep Learning on CPUs and GPUs
- ImageNet Training in Minutes
- Poseidon: A System Architecture for Efficient GPU-based Deep Learning on Multiple Machines
- Dual Path Networks
- Generative Adversarial Parallelization
- Sparse Tensor Algebra as a Parallel Programming Model
- AMPNet: Asynchronous Model-Parallel Training for Dynamic Neural Networks
- CHAOS: A Parallelization Scheme for Training Convolutional Neural Networks on Intel Xeon Phi
- Practical Block-wise Neural Network Architecture Generation
- Deep Learning at 15PF: Supervised and Semi-Supervised Classification for Scientific Data
- Communication-Optimal Convolutional Neural Nets
- Ease.ml: Towards Multi-tenant Resource Sharing for Machine Learning Workloads
- Parallel Multi Channel Convolution using General Matrix Multiplication
- SparCML: High-Performance Sparse Communication for Machine Learning
- On the Performance of Network Parallel Training in Artificial Neural Networks
- Asynchronous Parallel Stochastic Gradient Descent - A Numeric Core for Scalable Distributed Machine Learning Algorithms
- Distributed Hessian-Free Optimization for Deep Neural Network
- Instant Learning: Parallel Deep Neural Networks and Convolutional Bootstrapping
Cited by in corpus (37)
- Split learning for health: Distributed deep learning without sharing raw patient data
- Database Meets Deep Learning: Challenges and Opportunities
- Think Locally, Act Globally: Federated Learning with Local and Global Representations
- Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines
- Vector Symbolic Architectures as a Computing Framework for Emerging Hardware
- Communication optimization strategies for distributed deep neural network training: A survey
- Characterizing Deep-Learning I/O Workloads in TensorFlow
- Near-Optimal Sparse Allreduce for Distributed Deep Learning
- Taming Unbalanced Training Workloads in Deep Learning with Partial Collective Operations
- Augment your batch: better training with larger batches
- Flare: Flexible In-Network Allreduce
- Scalable Distributed DNN Training using TensorFlow and CUDA-Aware MPI: Characterization, Designs, and Performance Evaluation
- DataLens: Scalable Privacy Preserving Training via Gradient Compression and Aggregation
- Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems
- Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools
- Graph Processing on FPGAs: Taxonomy, Survey, Challenges
- Tighter Theory for Local SGD on Identical and Heterogeneous Data
- XPipe: Efficient Pipeline Model Parallelism for Multi-GPU DNN Training
- FPDeep: Scalable Acceleration of CNN Training on Deeply-Pipelined FPGA Clusters
- Union: An Automatic Workload Manager for Accelerating Network Simulation
- Gradient Descent with Compressed Iterates
- SparCML: High-Performance Sparse Communication for Machine Learning
- An Oracle for Guiding Large-Scale Model/Hybrid Parallel Training of Convolutional Neural Networks
- LayerPipe: Accelerating Deep Neural Network Training by Intra-Layer and Inter-Layer Gradient Pipelining and Multiprocessor Scheduling
- Efficient ConvNets for Analog Arrays
- Performance Analysis and Comparison of Distributed Machine Learning Systems
- Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Training
- Accelerating Neural Network Training with Distributed Asynchronous and Selective Optimization (DASO)
- Scavenger: A Cloud Service for Optimizing Cost and Performance of ML Training
- Speeding up Deep Learning with Transient Servers
- Characterizing and Modeling Distributed Training with Transient Cloud GPU Servers
- Stochastic Distributed Optimization for Machine Learning from Decentralized Features
- Sync-Switch: Hybrid Parameter Synchronization for Distributed Deep Learning
- ResIST: Layer-Wise Decomposition of ResNets for Distributed Training
- Pebbles, Graphs, and a Pinch of Combinatorics: Towards Tight I/O Lower Bounds for Statically Analyzable Programs
- Hydra: A System for Large Multi-Model Deep Learning
- A Random Gossip BMUF Process for Neural Language Modeling