492 citations · 1.5k across the 65 of their papers we have counts for
10 papers · 1 filter
Jenga: Effective Memory Management for Serving LLM with Heterogeneity
Chen Zhang, Kuntai Du, Shu Liu +10
Large language models (LLMs) are widely used but expensive to run, especially as inference workloads grow. To lower costs, maximizing the request batch size by managing GPU memory…
Mélange: Cost Efficient Large Language Model Serving by Exploiting GPU Heterogeneity
Tyler Griggs, Xiaoxuan Liu, Jiaxiang Yu +4
Large language models (LLMs) are increasingly integrated into many online services, yet they remain cost-prohibitive to deploy due to the requirement of expensive GPU instances. Pr…
MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving
Jiangfei Duan, Runyu Lu, Haojie Duanmu +5
Large language models (LLMs) have demonstrated remarkable performance, and organizations are racing to serve LLMs of varying sizes as endpoints for use-cases like chat, programming…
SuperServe: Fine-Grained Inference Serving for Unpredictable Workloads
Alind Khare, Dhruv Garg, Sukrit Kalra +3
The increasing deployment of ML models on the critical path of production applications in both datacenter and the edge requires ML inference serving systems to serve these models u…
NumS: Scalable Array Programming for the Cloud
Melih Elibol, Vinamra Benara, Samyu Yagati +4
Scientists increasingly rely on Python tools to perform scalable distributed memory array operations using rich, NumPy-like expressions. However, many of these tools rely on dynami…
HyperSched: Dynamic Resource Reallocation for Model Development on a Deadline
Richard Liaw, Romil Bhardwaj, Lisa Dunlap +4
Prior research in resource scheduling for machine learning training workloads has largely focused on minimizing job completion times. Commonly, these model training workloads colle…
The OoO VLIW JIT Compiler for GPU Inference
Paras Jain, Xiangxi Mo, Ajay Jain +3
Current trends in Machine Learning~(ML) inference on hardware accelerated devices (e.g., GPUs, TPUs) point to alarmingly low utilization. As ML inference is increasingly time-bound…
Dynamic Space-Time Scheduling for GPU Inference
Paras Jain, Xiangxi Mo, Ajay Jain +5
Serving deep neural networks in latency critical interactive settings often requires GPU acceleration. However, the small batch sizes typical in online inference results in poor GP…
InferLine: ML Prediction Pipeline Provisioning and Management for Tight Latency Objectives
Daniel Crankshaw, Gur-Eyal Sela, Corey Zumar +4
Serving ML prediction pipelines spanning multiple models and hardware accelerators is a key challenge in production machine learning. Optimally configuring these pipelines to meet…
Clipper: A Low-Latency Online Prediction Serving System
Daniel Crankshaw, Xin Wang, Giulio Zhou +3
Machine learning is being deployed in a growing number of applications which demand real-time, accurate, and robust predictions under heavy query load. However, most machine learni…