most citedSpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification

101 citations · 101 across the 1 of their papers we have counts for

collaborators

1 paper

cs.CL2023★ 101 cited

SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification

Xupeng Miao, Gabriele Oliaro, Zhihao Zhang +12

This paper introduces SpecInfer, a system that accelerates generative large language model (LLM) serving with tree-based speculative inference and verification. The key idea behind…