AIPerf: Automated machine learning as an AI-HPC benchmark

Abstract

The plethora of complex artificial intelligence (AI) algorithms and available high performance computing (HPC) power stimulates the expeditious development of AI components with heterogeneous designs. Consequently, the need for cross-stack performance benchmarking of AI-HPC systems emerges rapidly. The de facto HPC benchmark LINPACK can not reflect AI computing power and I/O performance without representative workload. The current popular AI benchmarks like MLPerf have fixed problem size therefore limited scalability. To address these issues, we propose an end-to-end benchmark suite utilizing automated machine learning (AutoML), which not only represents real AI scenarios, but also is auto-adaptively scalable to various scales of machines. We implement the algorithms in a highly parallel and flexible way to ensure the efficiency and optimization potential on diverse systems with customizable configurations. We utilize operations per second (OPS), which is measured in an analytical and systematic approach, as the major metric to quantify the AI performance. We perform evaluations on various systems to ensure the benchmark's stability and scalability, from 4 nodes with 32 NVIDIA Tesla T4 (56.1 Tera-OPS measured), up to 512 nodes with 4096 Huawei Ascend 910 (194.53 Peta-OPS measured), and the results show near-linear weak scalability. With flexible workload and single metric, our benchmark can scale and rank AI-HPC easily.

References in corpus (19)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Scikit-learn: Machine Learning in Python
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• Neural Architecture Search with Reinforcement Learning
• Neural Architecture Search: A Survey
• AutoML: A Survey of the State-of-the-Art
• Towards Automated Machine Learning: Evaluation and Comparison of AutoML Approaches and Tools
• Simple And Efficient Architecture Search for Convolutional Neural Networks
• Fathom: Reference Workloads for Modern Deep Learning Methods
• MLPerf Training Benchmark
• Net2Net: Accelerating Learning via Knowledge Transfer
• Convergence of Artificial Intelligence and High Performance Computing on NSF-supported Cyberinfrastructure
• TBD: Benchmarking and Analyzing Deep Neural Network Training
• AIBench: An Industry Standard Internet Service AI Benchmark Suite
• HPC AI500: The Methodology, Tools, Roofline Performance Models, and Metrics for Benchmarking HPC AI Systems
• AI Matrix: A Deep Learning Benchmark for Alibaba Data Centers
• Benchmarking the Performance and Energy Efficiency of AI Accelerators for AI Training