MISO: Exploiting Multi-Instance GPU Capability on Multi-Tenant Systems for Machine Learning

Abstract

GPU technology has been improving at an expedited pace in terms of size and performance, empowering HPC and AI/ML researchers to advance the scientific discovery process. However, this also leads to inefficient resource usage, as most GPU workloads, including complicated AI/ML models, are not able to utilize the GPU resources to their fullest extent -- encouraging support for GPU multi-tenancy. We propose MISO, a technique to exploit the Multi-Instance GPU (MIG) capability on the latest NVIDIA datacenter GPUs (e.g., A100, H100) to dynamically partition GPU resources among co-located jobs. MISO's key insight is to use the lightweight, more flexible Multi-Process Service (MPS) capability to predict the best MIG partition allocation for different jobs, without incurring the overhead of implementing them during exploration. Due to its ability to utilize GPU resources more efficiently, MISO achieves 49% and 16% lower average job completion time than the unpartitioned and optimal static GPU partition schemes, respectively.

References in corpus (6)

• Graph Neural Networks in Particle Physics
• Characterization and Prediction of Deep Learning Workloads in Large-Scale GPU Datacenters
• Serving DNNs like Clockwork: Performance Predictability from the Bottom Up
• Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads
• Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloads
• MAPA: Multi-Accelerator Pattern Allocation Policy for Multi-Tenant GPU Servers

Cited by in corpus (5)

• Clover: Toward Sustainable AI with Carbon-Aware Machine Learning Inference Service
• Hierarchical Resource Partitioning on Modern GPUs: A Reinforcement Learning Approach
• KAIROS: Building Cost-Efficient Machine Learning Inference Systems with Heterogeneous Cloud Resources
• Energy Efficient Scheduling of AI/ML Workloads on Multi Instance GPUs with Dynamic Repartitioning
• A comprehensive evaluation of spatial co-execution on GPUs using MPS and MIG technologies