4 papers
Energy-Aware Scheduling for Serverless LLM Serving on Shared GPUs
Tianyu Wang, Gourav Rattihalli, Aditya Dhakal +2
As LLM inference becomes a major cloud workload, its growing energy footprint makes cluster-wide energy optimization increasingly important. Serverless LLM serving helps platforms…
Predict, Reuse, and Repair: Accelerating Dynamic Sparse Attention for Long-Context LLM Decoding
Tianyu Wang, Gourav Rattihalli, Aditya Dhakal +4
Dynamic sparse attention (DSA) accelerates long-context LLM decoding by attending to only the top-K KV blocks relevant to each query, but it introduces a serialized selection-to-at…
Improving GPU Multi-Tenancy Through Dynamic Multi-Instance GPU Reconfiguration
Tianyu Wang, Sheng Li, Bingyao Li +6
Continuous learning (CL) has emerged as one of the most popular deep learning paradigms deployed in modern cloud GPUs. Specifically, CL has the capability to continuously update th…
Improving Multi-Instance GPU Efficiency via Sub-Entry Sharing TLB Design
Bingyao Li, Yueqi Wang, Tianyu Wang +4
NVIDIA's Multi-Instance GPU (MIG) technology enables partitioning GPU computing power and memory into separate hardware instances, providing complete isolation including compute re…