AMPNet: Asynchronous Model-Parallel Training for Dynamic Neural Networks

Abstract

New types of machine learning hardware in development and entering the market hold the promise of revolutionizing deep learning in a manner as profound as GPUs. However, existing software frameworks and training algorithms for deep learning have yet to evolve to fully leverage the capability of the new wave of silicon. We already see the limitations of existing algorithms for models that exploit structured input via complex and instance-dependent control flow, which prohibits minibatching. We present an asynchronous model-parallel (AMP) training algorithm that is specifically motivated by training on networks of interconnected devices. Through an implementation on multi-core CPUs, we show that AMP training converges to the same accuracy as conventional synchronous training algorithms in a similar number of epochs, but utilizes the available hardware more efficiently even for small minibatch sizes, resulting in significantly shorter overall training times. Our framework opens the door for scaling up a new class of deep learning models that cannot be efficiently trained today.

17 pages, 13 figures

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• Gated Graph Sequence Neural Networks
• Adaptive Computation Time for Recurrent Neural Networks
• DyNet: The Dynamic Neural Network Toolkit
• Deep Networks with Stochastic Depth
• Neural Programmer-Interpreters
• Recurrent Dropout without Memory Loss
• Deep Learning with Dynamic Computation Graphs
• Decoupled Neural Interfaces using Synthetic Gradients
• Hierarchical Memory Networks
• A Fast Unified Model for Parsing and Sentence Understanding
• Spatially Adaptive Computation Time for Residual Networks

Cited by in corpus (10)

• Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysis
• Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines
• XPipe: Efficient Pipeline Model Parallelism for Multi-GPU DNN Training
• Declarative Recursive Computation on an RDBMS, or, Why You Should Use a Database For Distributed Machine Learning
• Local Critic Training for Model-Parallel Learning of Deep Neural Networks
• Orchestrating the Development Lifecycle of Machine Learning-Based IoT Applications: A Taxonomy and Survey
• Pipelined Backpropagation at Scale: Training Large Models without Batches
• Automatic Graph Partitioning for Very Large-scale Deep Learning
• DistIR: An Intermediate Representation and Simulator for Efficient Neural Network Distribution
• Gear Training: A new way to implement high-performance model-parallel training