101 citations · 162 across the 6 of their papers we have counts for
Showing 2023Show all
2 papers · 1 filter
cs.DC2023★ 3 cited
SpotServe: Serving Generative Large Language Models on Preemptible Instances
Xupeng Miao, Chunan Shi, Jiangfei Duan +4
The high computational and memory requirements of generative large language models (LLMs) make it challenging to serve them cheaply. This paper aims to reduce the monetary cost for…
cs.CL2023★ 101 cited
SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang +12
This paper introduces SpecInfer, a system that accelerates generative large language model (LLM) serving with tree-based speculative inference and verification. The key idea behind…