MLPerf Training Benchmark

Abstract

Machine learning (ML) needs industry-standard performance benchmarks to support design and competitive evaluation of the many emerging software and hardware solutions for ML. But ML training presents three unique benchmarking challenges absent from other domains: optimizations that improve training throughput can increase the time to solution, training is stochastic and time to solution exhibits high variance, and software and hardware systems are so diverse that fair benchmarking with the same binary, code, and even hyperparameters is difficult. We therefore present MLPerf, an ML benchmark that overcomes these challenges. Our analysis quantitatively evaluates MLPerf's efficacy at driving performance and scalability improvements across two rounds of results from multiple vendors.

MLSys 2020

References in corpus (16)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Playing Atari with Deep Reinforcement Learning
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• cuDNN: Efficient Primitives for Deep Learning
• One weird trick for parallelizing convolutional neural networks
• Trained Ternary Quantization
• The Loss Surfaces of Multilayer Networks
• Large Batch Training of Convolutional Networks
• Deep Learning Recommendation Model for Personalization and Recommendation Systems
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes
• Fathom: Reference Workloads for Modern Deep Learning Methods
• Image Classification at Supercomputer Scale
• Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash
• Optimizing Network Performance for Distributed DNN Training on GPU Clusters: ImageNet/AlexNet Training in 1.5 Minutes
• Scalable Realistic Recommendation Datasets through Fractal Expansions

Cited by in corpus (41)

• Review: Deep Learning in Electron Microscopy
• A Survey on Deep Learning Hardware Accelerators for Heterogeneous HPC Platforms
• Understanding Data Storage and Ingestion for Large-Scale Deep Recommendation Model Training
• QUARK: A Framework for Quantum Computing Application Benchmarking
• DeepRecSys: A System for Optimizing End-To-End At-scale Neural Recommendation Inference
• MoCA: Memory-Centric, Adaptive Execution for Multi-Tenant Deep Neural Networks
• Developing a Recommendation Benchmark for MLPerf Training and Inference
• On the Utility of Gradient Compression in Distributed Training Systems
• HULK: An Energy Efficiency Benchmark Platform for Responsible Natural Language Processing
• The Architectural Implications of Facebook's DNN-based Personalized Recommendation
• TT-Rec: Tensor Train Compression for Deep Learning Recommendation Models
• HPC AI500: The Methodology, Tools, Roofline Performance Models, and Metrics for Benchmarking HPC AI Systems
• Exploring the limits of Concurrency in ML Training on Google TPUs
• User Response Prediction in Online Advertising
• tf.data: A Machine Learning Data Processing Framework
• Plumber: Diagnosing and Removing Performance Bottlenecks in Machine Learning Data Pipelines
• Hardware-accelerated Simulation-based Inference of Stochastic Epidemiology Models for COVID-19
• Understanding Training Efficiency of Deep Learning Recommendation Models at Scale
• A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes
• MLHarness: A Scalable Benchmarking System for MLCommons
• BenchCouncil's View on Benchmarking AI and Other Emerging Workloads
• Multi-node Bert-pretraining: Cost-efficient Approach
• Sketchy Empirical Natural Gradient Methods for Deep Learning
• Arithmetic-Intensity-Guided Fault Tolerance for Neural Network Inference on GPUs
• ML-EXray: Visibility into ML Deployment on the Edge
• Jointly Optimizing Preprocessing and Inference for DNN-based Visual Analytics
• CPR: Understanding and Improving Failure Tolerant Training for Deep Learning Recommendation with Partial Recovery
• A Generalizable Approach to Learning Optimizers
• AIBench Scenario: Scenario-distilling AI Benchmarking
• AIBench Training: Balanced Industry-Standard AI Training Benchmarking
• GEVO: GPU Code Optimization using Evolutionary Computation
• Concurrent Adversarial Learning for Large-Batch Training
• DLSpec: A Deep Learning Task Exchange Specification
• Neural Model-based Optimization with Right-Censored Observations
• The Pitfall of Evaluating Performance on Emerging AI Accelerators
• AIPerf: Automated machine learning as an AI-HPC benchmark
• AIBench: An Agile Domain-specific Benchmarking Methodology and an AI Benchmark Suite
• Machine Learning Systems for Intelligent Services in the IoT: A Survey
• NG+ : A Multi-Step Matrix-Product Natural Gradient Method for Deep Learning
• Bosch Deep Learning Hardware Benchmark
• Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scale