Characterization and Prediction of Deep Learning Workloads in Large-Scale GPU Datacenters

Abstract

Modern GPU datacenters are critical for delivering Deep Learning (DL) models and services in both the research community and industry. When operating a datacenter, optimization of resource scheduling and management can bring significant financial benefits. Achieving this goal requires a deep understanding of the job features and user behaviors. We present a comprehensive study about the characteristics of DL jobs and resource management. First, we perform a large-scale analysis of real-world job traces from SenseTime. We uncover some interesting conclusions from the perspectives of clusters, jobs and users, which can facilitate the cluster system designs. Second, we introduce a general-purpose framework, which manages resources based on historical data. As case studies, we design: a Quasi-Shortest-Service-First scheduling service, which can minimize the cluster-wide average job completion time by up to 6.5x; and a Cluster Energy Saving service, which improves overall cluster utilization by up to 13%.

arXiv commentThis paper has been accepted by the International Conference for High Performance Computing, Networking, Storage, and Analysis (SC21), Nov 14-19, 2021, St. Louis, USA

References in corpus (7)

• Language Models are Few-Shot Learners
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Serverless in the Wild: Characterizing and Optimizing the Serverless Workload at a Large Cloud Provider
• Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads
• Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloads
• A Workload Analysis of NSF's Innovative HPC Resources Using XDMoD
• GPGPU Performance Estimation with Core and Memory Frequency Scaling

Cited by in corpus (6)

• Task Placement and Resource Allocation for Edge Machine Learning: A GNN-based Multi-Agent Reinforcement Learning Paradigm
• MISO: Exploiting Multi-Instance GPU Capability on Multi-Tenant Systems for Machine Learning
• Toward Sustainable HPC: Carbon Footprint Estimation and Environmental Implications of HPC Systems
• FuncPipe: A Pipelined Serverless Framework for Fast and Cost-efficient Training of Deep Learning Models
• An Empirical Characterization of Outages and Incidents in Public Services for Large Language Models
• Power- and Fragmentation-aware Online Scheduling for GPU Datacenters