AIBench: An Industry Standard Internet Service AI Benchmark Suite

Abstract

Today's Internet Services are undergoing fundamental changes and shifting to an intelligent computing era where AI is widely employed to augment services. In this context, many innovative AI algorithms, systems, and architectures are proposed, and thus the importance of benchmarking and evaluating them rises. However, modern Internet services adopt a microservice-based architecture and consist of various modules. The diversity of these modules and complexity of execution paths, the massive scale and complex hierarchy of datacenter infrastructure, the confidential issues of data sets and workloads pose great challenges to benchmarking. In this paper, we present the first industry-standard Internet service AI benchmark suite---AIBench with seventeen industry partners, including several top Internet service providers. AIBench provides a highly extensible, configurable, and flexible benchmark framework that contains loosely coupled modules. We identify sixteen prominent AI problem domains like learning to rank, each of which forms an AI component benchmark, from three most important Internet service domains: search engine, social network, and e-commerce, which is by far the most comprehensive AI benchmarking effort. On the basis of the AIBench framework, abstracting the real-world data sets and workloads from one of the top e-commerce providers, we design and implement the first end-to-end Internet service AI benchmark, which contains the primary modules in the critical paths of an industry scale application and is scalable to deploy on different cluster scales. The specifications, source code, and performance numbers are publicly available from the benchmark council web site http://www.benchcouncil.org/AIBench/index.html.

arXiv comment24 pages

References in corpus (11)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
• Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision
• Fathom: Reference Workloads for Modern Deep Learning Methods
• TensorFlow-Serving: Flexible, High-Performance ML Serving
• BigDataBench: A Scalable and Unified Big Data and AI Benchmark Suite
• Frustrated with Replicating Claims of a Shared Model? A Solution

Cited by in corpus (9)

• Machine Learning and Big Scientific Data
• HPC AI500: The Methodology, Tools, Roofline Performance Models, and Metrics for Benchmarking HPC AI Systems
• The Design and Implementation of a Scalable DL Benchmarking Platform
• BenchCouncil's View on Benchmarking AI and Other Emerging Workloads
• Benchmarking the Performance and Energy Efficiency of AI Accelerators for AI Training
• AI Tax: The Hidden Cost of AI Data Center Applications
• AIBench Training: Balanced Industry-Standard AI Training Benchmarking
• AIPerf: Automated machine learning as an AI-HPC benchmark
• HPC AI500: A Benchmark Suite for HPC AI Systems