HPC AI500: Representative, Repeatable and Simple HPC AI Benchmarking
arXiv:2102.12848
Abstract
Recent years witness a trend of applying large-scale distributed deep learning algorithms (HPC AI) in both business and scientific computing areas, whose goal is to speed up the training time to achieve a state-of-the-art quality. The HPC AI benchmarks accelerate the process. Unfortunately, benchmarking HPC AI systems at scale raises serious challenges. This paper presents a representative, repeatable and simple HPC AI benchmarking methodology. Among the seventeen AI workloads of AIBench Training -- by far the most comprehensive AI Training benchmarks suite -- we choose two representative and repeatable AI workloads. The selected HPC AI benchmarks include both business and scientific computing: Image Classification and Extreme Weather Analytics. To rank HPC AI systems, we present a new metric named Valid FLOPS, emphasizing both throughput performance and a target quality. The specification, source code, datasets, and HPC AI500 ranking numbers are publicly available from \url{https://www.benchcouncil.org/HPCAI500/}.
arXiv admin note: substantial text overlap with arXiv:2007.00279
References in corpus (4)
- Extremely Large Minibatch SGD: Training ResNet-50 on ImageNet in 15 Minutes
- Comparison and Benchmarking of AI Models and Frameworks on Mobile Devices
- HPC AI500: The Methodology, Tools, Roofline Performance Models, and Metrics for Benchmarking HPC AI Systems
- Scaling GRPC Tensorflow on 512 nodes of Cori Supercomputer