27 citations · 34 across the 7 of their papers we have counts for
5 papers · 1 filter
Jenga: Effective Memory Management for Serving LLM with Heterogeneity
Chen Zhang, Kuntai Du, Shu Liu +10
Large language models (LLMs) are widely used but expensive to run, especially as inference workloads grow. To lower costs, maximizing the request batch size by managing GPU memory…
SkyStore: Cost-Optimized Object Storage Across Regions and Clouds
Shu Liu, Xiangxi Mo, Moshik Hershcovitch +11
Modern applications span multiple clouds to reduce costs, avoid vendor lock-in, and leverage low-availability resources in another cloud. However, standard object stores operate wi…
The OoO VLIW JIT Compiler for GPU Inference
Paras Jain, Xiangxi Mo, Ajay Jain +3
Current trends in Machine Learning~(ML) inference on hardware accelerated devices (e.g., GPUs, TPUs) point to alarmingly low utilization. As ML inference is increasingly time-bound…
Dynamic Space-Time Scheduling for GPU Inference
Paras Jain, Xiangxi Mo, Ajay Jain +5
Serving deep neural networks in latency critical interactive settings often requires GPU acceleration. However, the small batch sizes typical in online inference results in poor GP…
InferLine: ML Prediction Pipeline Provisioning and Management for Tight Latency Objectives
Daniel Crankshaw, Gur-Eyal Sela, Corey Zumar +4
Serving ML prediction pipelines spanning multiple models and hardware accelerators is a key challenge in production machine learning. Optimally configuring these pipelines to meet…