1 citations · 1 across the 9 of their papers we have counts for
9 papers
Less Experts, Faster Decoding: Cost-Aware Speculative Decoding for Mixture-of-Experts
Jincheng Xie, Runheng Liu, Heyan Huang +4
Sparse Mixture-of-Experts (MoE) models have become an important approach for scaling Large Language Models (LLMs), but their inference efficiency depends strongly on expert activat…
AdaPLD: Adaptive Retrieval and Reuse for Efficient Model-Free Speculative Decoding
Runheng Liu, Jincheng Xie, Wen Hu +2
Speculative decoding accelerates generation by verifying multiple drafted tokens in a single target-model forward pass, reducing sequential decoding iterations. Model-free variants…
SPECTRE: Hybrid Ordinary-Parallel Speculative Serving for Resource-Efficient LLM Inference
Jincheng Xie, Yawen Ling, Qi Xiao +4
LLM serving platforms are increasingly deployed as multi-model cloud systems, where user demand is often long-tailed: a few popular large models receive most requests, while many s…
SAT-BO: Verification Rule Learning and Optimization for FraudTransaction Detection
Mao Luo, Zhi Wang, Yiwen Huang +5
Electronic payment platforms are estimated to process billions oftransactions daily, with the cumulative value of these transactionspotentially reaching into the trillions. Even a…
CAKE: Cascading and Adaptive KV Cache Eviction with Layer Preferences
Ziran Qin, Yuchen Cao, Mingbao Lin +5
Large language models (LLMs) excel at processing long sequences, boosting demand for key-value (KV) caching. While recent efforts to evict KV cache have alleviated the inference bu…
SCOOT: SLO-Oriented Performance Tuning for LLM Inference Engines
Ke Cheng, Zhi Wang, Wen Hu +3
As large language models (LLMs) are gaining increasing popularity across a wide range of web applications, it is of great importance to optimize service-level objectives (SLOs) for…
Enabling Efficient Batch Serving for LMaaS via Generation Length Prediction
Ke Cheng, Wen Hu, Zhi Wang +3
Nowadays, large language models (LLMs) are published as a service and can be accessed by various applications via APIs, also known as language-model-as-a-service (LMaaS). Without k…
Slice-Level Scheduling for High Throughput and Load Balanced LLM Serving
Ke Cheng, Wen Hu, Zhi Wang +3
Large language models (LLMs) iteratively generate text token by token, with memory usage increasing with the length of generated token sequences. Since the request generation lengt…
VDTuner: Automated Performance Tuning for Vector Data Management Systems
Tiannuo Yang, Wen Hu, Wangqi Peng +4
Vector data management systems (VDMSs) have become an indispensable cornerstone in large-scale information retrieval and machine learning systems like large language models. To enh…