4 citations · 6 across the 7 of their papers we have counts for
8 papers
Energy-Aware Scheduling for Serverless LLM Serving on Shared GPUs
Tianyu Wang, Gourav Rattihalli, Aditya Dhakal +2
As LLM inference becomes a major cloud workload, its growing energy footprint makes cluster-wide energy optimization increasingly important. Serverless LLM serving helps platforms…
Predict, Reuse, and Repair: Accelerating Dynamic Sparse Attention for Long-Context LLM Decoding
Tianyu Wang, Gourav Rattihalli, Aditya Dhakal +4
Dynamic sparse attention (DSA) accelerates long-context LLM decoding by attending to only the top-K KV blocks relevant to each query, but it introduces a serialized selection-to-at…
Accelerating Physical Property Reasoning for Augmented Visual Cognition
Hongbo Lan, Zhenlin An, Haoyu Li +2
This paper introduces \sysname, a system that accelerates vision-guided physical property reasoning to enable augmented visual cognition. \sysname minimizes the run-time latency of…
LLM-dCache: Improving Tool-Augmented LLMs with GPT-Driven Localized Data Caching
Simranjit Singh, Michael Fore, Andreas Karatzas +6
As Large Language Models (LLMs) broaden their capabilities to manage thousands of API calls, they are confronted with complex data operations across vast datasets with significant…
A Survey of Multi-Tenant Deep Learning Inference on GPU
Fuxun Yu, Di Wang, Longfei Shangguan +3
Deep Learning (DL) models have achieved superior performance. Meanwhile, computing hardware like NVIDIA GPUs also demonstrated strong computing scaling trends with 2x throughput an…
Automated Runtime-Aware Scheduling for Multi-Tenant DNN Inference on GPU
Fuxun Yu, Shawn Bray, Di Wang +4
With the fast development of deep neural networks (DNNs), many real-world applications are adopting multiple models to conduct compound tasks, such as co-running classification, de…
A Survey of Large-Scale Deep Learning Serving System Optimization: Challenges and Opportunities
Fuxun Yu, Di Wang, Longfei Shangguan +4
Deep Learning (DL) models have achieved superior performance in many application domains, including vision, language, medical, commercial ads, entertainment, etc. With the fast dev…
Towards Latency-aware DNN Optimization with GPU Runtime Analysis and Tail Effect Elimination
Fuxun Yu, Zirui Xu, Tong Shen +12
Despite the superb performance of State-Of-The-Art (SOTA) DNNs, the increasing computational cost makes them very challenging to meet real-time latency and accuracy requirements. A…