ChainerMN: Scalable Distributed Deep Learning Framework

Abstract

One of the keys for deep learning to have made a breakthrough in various fields was to utilize high computing powers centering around GPUs. Enabling the use of further computing abilities by distributed processing is essential not only to make the deep learning bigger and faster but also to tackle unsolved challenges. We present the design, implementation, and evaluation of ChainerMN, the distributed deep learning framework we have developed. We demonstrate that ChainerMN can scale the learning process of the ResNet-50 model to the ImageNet dataset up to 128 GPUs with the parallel efficiency of 90%.

References in corpus (4)

• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Revisiting Distributed Synchronous SGD
• Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes

Cited by in corpus (18)

• Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes
• BigDL: A Distributed Deep Learning Framework for Big Data
• Train Sparsely, Generate Densely: Memory-efficient Unsupervised Training of High-resolution Temporal GAN
• Neural Tangents: Fast and Easy Infinite Neural Networks in Python
• Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools
• Deep Leakage from Gradients
• Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networks
• Classifying near-threshold enhancement using deep neural network
• PFDet: 2nd Place Solution to Open Images Challenge 2018 Object Detection Track
• An Oracle for Guiding Large-Scale Model/Hybrid Parallel Training of Convolutional Neural Networks
• SAPAG: A Self-Adaptive Privacy Attack From Gradients
• A(DP)${}^2$SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy
• Low-latency job scheduling with preemption for the development of deep learning
• Deep Gated Multi-modal Learning: In-hand Object Pose Changes Estimation using Tactile and Image Data
• Hitachi at MRP 2019: Unified Encoder-to-Biaffine Network for Cross-Framework Meaning Representation Parsing
• Gradient Noise Convolution (GNC): Smoothing Loss Function for Distributed Large-Batch SGD
• Team PFDet's Methods for Open Images Challenge 2019
• Disaggregated Accelerator Management System for Cloud Data Centers