6 papers
CacheSmith: Accurate RAG Serving under Latency SLOs with Adaptive KV Reuse
Xinyue Ma, Damin Lee, Aashaka Shah +1
This paper presents CacheSmith, a KV cache reuse system for RAG prefill that maximizes quality-gated SLO attainment. While context caching lowers prefill latency by reusing precomp…
DIALER: A Case for Improving Rare-Class Accuracy in Retraining-Free Edge Video Analytics
Dongyoon Ryu, Sungho Jeon, Xinyue Ma +4
Edge video analytics with lightweight models is prone to accuracy degradation due to persistent distributional shifts in live video streams. While continuous learning (CL) addresse…
Heddle: Learning Structural Templates for Parallelism Planning on Heterogeneous GPU Clusters
Taeyoon Kim, Yonguk Song, Seoyeong Choy +4
Training large machine learning models on shared GPU infrastructures faces two challenges: (1) GPU availability shifts dynamically with varying resource demands from tenants, and (…
Heterogeneous LLM Serving with General-Purpose Processing-Near-Memory for Retrieval-Based Sparse Attention
Hyungkyu Ham, Junhyeong Bae, Seungheon Lee +2
This paper presents a heterogeneous decode-phase serving system that relocates the KV cache out of GPU memory, motivated by the retrieval-based sparse attention that recent frontie…
OrbitFlow: SLO-Aware Long-Context LLM Serving with Fine-Grained KV Cache Reconfiguration
Xinyue Ma, Heelim Hong, Taegeon Um +4
Serving long-context LLMs is challenging because request lengths and batch composition vary during token generation, causing the memory footprint to fluctuate significantly at runt…
REP: Resource-Efficient Prompting for Rehearsal-Free Continual Learning
Sungho Jeon, Xinyue Ma, Kwang In Kim +1
Recent rehearsal-free continual learning (CL) methods guided by prompts achieve strong performance on vision tasks with non-stationary data but remain resource-intensive, hindering…