AIBench Training: Balanced Industry-Standard AI Training Benchmarking

Abstract

Earlier-stage evaluations of a new AI architecture/system need affordable benchmarks. Only using a few AI component benchmarks like MLPerfalone in the other stages may lead to misleading conclusions. Moreover, the learning dynamics are not well understood, and the benchmarks' shelf-life is short. This paper proposes a balanced benchmarking methodology. We use real-world benchmarks to cover the factors space that impacts the learning dynamics to the most considerable extent. After performing an exhaustive survey on Internet service AI domains, we identify and implement nineteen representative AI tasks with state-of-the-art models. For repeatable performance ranking (RPR subset) and workload characterization (WC subset), we keep two subsets to a minimum for affordability. We contribute by far the most comprehensive AI training benchmark suite. The evaluations show: (1) AIBench Training (v1.1) outperforms MLPerfTraining (v0.7) in terms of diversity and representativeness of model complexity, computational cost, convergent rate, computation, and memory access patterns, and hotspot functions; (2) Against the AIBench full benchmarks, its RPR subset shortens the benchmarking cost by 64%, while maintaining the primary workload characteristics; (3) The performance ranking shows the single-purpose AI accelerator like TPU with the optimized TensorFlowframework performs better than that of GPUs while losing the latter's general support for various AI models. The specification, source code, and performance numbers are available from the AIBench homepage https://www.benchcouncil.org/aibench-training/index.html.

arXiv commentISPASS 2021

References in corpus (20)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• Neural Architecture Search: A Survey
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
• Benchmark Analysis of Representative Deep Neural Network Architectures
• Efficient Neural Architecture Search via Parameter Sharing
• Deep Learning Recommendation Model for Personalization and Recommendation Systems
• Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond
• Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision
• Fathom: Reference Workloads for Modern Deep Learning Methods
• MLPerf Training Benchmark
• TBD: Benchmarking and Analyzing Deep Neural Network Training
• AIBench: An Industry Standard Internet Service AI Benchmark Suite
• BigDataBench: A Scalable and Unified Big Data and AI Benchmark Suite
• Perceive Your Users in Depth: Learning Universal User Representations from Multiple E-commerce Tasks
• HPC AI500: Representative, Repeatable and Simple HPC AI Benchmarking
• AIBench Scenario: Scenario-distilling AI Benchmarking

Cited by in corpus (1)

• HPC AI500: The Methodology, Tools, Roofline Performance Models, and Metrics for Benchmarking HPC AI Systems