AIBench Scenario: Scenario-distilling AI Benchmarking
arXiv:2005.03459
Abstract
Modern real-world application scenarios like Internet services consist of a diversity of AI and non-AI modules with huge code sizes and long and complicated execution paths, which raises serious benchmarking or evaluating challenges. Using AI components or micro benchmarks alone can lead to error-prone conclusions. This paper presents a methodology to attack the above challenge. We formalize a real-world application scenario as a Directed Acyclic Graph-based model and propose the rules to distill it into a permutation of essential AI and non-AI tasks, which we call a scenario benchmark. Together with seventeen industry partners, we extract nine typical scenario benchmarks. We design and implement an extensible, configurable, and flexible benchmark framework. We implement two Internet service AI scenario benchmarks based on the framework as proxies to two real-world application scenarios. We consider scenario, component, and micro benchmarks as three indispensable parts for evaluating. Our evaluation shows the advantage of our methodology against using component or micro AI benchmarks alone. The specifications, source code, testbed, and results are publicly available from \url{https://www.benchcouncil.org/aibench/scenario/}.
arXiv commentThis paper has been accepted by The 30th International Conference on Parallel Architectures and Compilation Techniques (PACT 2021)
References in corpus (18)
- Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
- TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
- Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
- Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
- Wasserstein GAN
- Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision
- Bao: Learning to Steer Query Optimizers
- Fathom: Reference Workloads for Modern Deep Learning Methods
- MLPerf Training Benchmark
- TensorFlow-Serving: Flexible, High-Performance ML Serving
- TBD: Benchmarking and Analyzing Deep Neural Network Training
- BigDataBench: A Scalable and Unified Big Data and AI Benchmark Suite
- Perceive Your Users in Depth: Learning Universal User Representations from Multiple E-commerce Tasks
- Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender System
- BenchCouncil's View on Benchmarking AI and Other Emerging Workloads
- Data Motifs: A Lens Towards Fully Understanding Big Data and AI Workloads
- MLModelScope: A Distributed Platform for Model Evaluation and Benchmarking at Scale
- MLPerf Tiny Benchmark