3 citations · 3 across the 3 of their papers we have counts for
5 papers
Simulate Before You Scale: High-Fidelity LLM Inference Planning Without GPUs
Dipanwita Guhathakurta, Mert Toslali, Srinivasan Parthasarathy +6
Deploying large language model (LLM) inference at scale requires jointly selecting hardware layout, parallelism strategy, scheduling policy, admission control, and request routing…
An Approximate Queueing Model of LLM Inference Serving for SLO-Driven Autoscaling
Vishakha Ramani, Asser N. Tantawi
Performance models of LLM servers support both latency evaluation and the design of controllers for autoscaling against service level objectives (SLOs) and for inference optimizati…
SMART-MIG: A Learning Framework for Scalable and Energy-Efficient GPU Scheduling
Wenqing Yu, Neel Karia, Tanvi Hisaria +3
The emergence of Multi-Instance GPU (MIG) technology enables us to run smaller machine learning models on partitions of a GPU rather than the entire device, thus improving utilizat…
Energy Efficient Scheduling of AI/ML Workloads on Multi Instance GPUs with Dynamic Repartitioning
Ellie Lipe, Neel Karia, Connor Espenshade +3
Increasing demand from AI/ML workloads is exacerbating the rising energy consumption of data centers. Recent advances in hardware such as NVIDIA's Multi Instance GPUs (MIGs) offer…
WVA: A Global Optimization Control Plane for llmd
Abhishek Malvankar, Lionel Villard, Mohammed Abdi +6
As Large Language Models (LLMs) scale to handle massive concurrent traffic, optimizing the infrastructure required for inference has become a primary challenge. To manage the high…