Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads

Abstract

With widespread advances in machine learning, a number of large enterprises are beginning to incorporate machine learning models across a number of products. These models are typically trained on shared, multi-tenant GPU clusters. Similar to existing cluster computing workloads, scheduling frameworks aim to provide features like high efficiency, resource isolation, fair sharing across users, etc. However Deep Neural Network (DNN) based workloads, predominantly trained on GPUs, differ in two significant ways from traditional big data analytics workloads. First, from a cluster utilization perspective, GPUs represent a monolithic resource that cannot be shared at a fine granularity across users. Second, from a workload perspective, deep learning frameworks require gang scheduling reducing the flexibility of scheduling and making the jobs themselves inelastic to failures at runtime. In this paper we present a detailed workload characterization of a two-month long trace from a multi-tenant GPU cluster in a large enterprise. By correlating scheduler logs with logs from individual jobs, we study three distinct issues that affect cluster utilization for DNN training workloads on multi-tenant clusters: (1) the effect of gang scheduling and locality constraints on queuing, (2) the effect of locality on GPU utilization, and (3) failures during training. Based on our experience running a large-scale operation, we provide design guidelines pertaining to next-generation cluster schedulers for DNN training workloads.

References in corpus (5)

• TensorFlow: A system for large-scale machine learning
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• A Reliable Effective Terascale Linear Learning System
• DeepProf: Performance Analysis for Deep Learning Applications via Mining GPU Execution Patterns

Cited by in corpus (13)

• Characterization and Prediction of Deep Learning Workloads in Large-Scale GPU Datacenters
• MISO: Exploiting Multi-Instance GPU Capability on Multi-Tenant Systems for Machine Learning
• Toward Sustainable HPC: Carbon Footprint Estimation and Environmental Implications of HPC Systems
• Accelerating Recommendation System Training by Leveraging Popular Choices
• Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques and Tools
• Reducing Energy Bloat in Large Model Training
• Communication Contention Aware Scheduling of Multiple Deep Learning Training Jobs
• Communication-efficient Decentralized Machine Learning over Heterogeneous Networks
• Literature Review and Implementation Overview: High Performance Computing with Graphics Processing Units for Classroom and Research Use
• Machine Learning Systems for Intelligent Services in the IoT: A Survey
• Elastic deep learning in multi-tenant GPU cluster
• Online Evolutionary Batch Size Orchestration for Scheduling Deep Learning Workloads in GPU Clusters
• The MIT Supercloud Dataset