3 citations · 6 across the 17 of their papers we have counts for
Showing cs.AIShow all
2 papers · 1 filter
cs.AI2024
TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodput
Xiaoxuan Liu, Jongseok Park, Langxiang Hu +10
Large Language Model (LLM) serving systems batch concurrent user requests to achieve efficient serving. However, in real-world deployments, such inter-request parallelism from batc…
cs.AI2023★ 3 cited
Online Speculative Decoding
Xiaoxuan Liu, Lanxiang Hu, Peter Bailis +4
Speculative decoding is a pivotal technique to accelerate the inference of large language models (LLMs) by employing a smaller draft model to predict the target model's outputs. Ho…