Scale MLPerf-0.6 models on Google TPU-v3 Pods
arXiv:1909.09756
Abstract
The recent submission of Google TPU-v3 Pods to the industry wide MLPerf v0.6 training benchmark demonstrates the scalability of a suite of industry relevant ML models. MLPerf defines a suite of models, datasets and rules to follow when benchmarking to ensure results are comparable across hardware, frameworks and companies. Using this suite of models, we discuss the optimizations and techniques including choice of optimizer, spatial partitioning and weight update sharding necessary to scale to 1024 TPU chips. Furthermore, we identify properties of models that make scaling them challenging, such as limited data parallelism and unscaled weights. These optimizations contribute to record performance in transformer, Resnet-50 and SSD in the Google MLPerf-0.6 submission.
References in corpus (2)
Cited by in corpus (16)
- Learning to Simulate Complex Physics with Graph Networks
- Tesseract: Parallelize the Tensor Parallelism Efficiently
- Automatic Cross-Replica Sharding of Weight Update in Data-Parallel Training
- RetinaTrack: Online Single Stage Joint Detection and Tracking
- XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages
- Improving 3D Object Detection through Progressive Population Based Augmentation
- KAISA: An Adaptive Second-Order Optimizer Framework for Deep Neural Networks
- Exploring the limits of Concurrency in ML Training on Google TPUs
- PERF-Net: Pose Empowered RGB-Flow Net
- A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes
- Improving Robustness using Generated Data
- Demystifying BERT: Implications for Accelerator Design
- Loss Landscape Dependent Self-Adjusting Learning Rates in Decentralized Stochastic Gradient Descent
- Asynchronous Decentralized Distributed Training of Acoustic Models
- Highly Available Data Parallel ML training on Mesh Networks
- Progressive Compressed Records: Taking a Byte out of Deep Learning Data