7 papers
When to Reason: Semantic Router for vLLM
Chen Wang, Xunzhuo Liu, Yuhan Liu +4
Large Language Models (LLMs) demonstrate substantial accuracy gains when augmented with reasoning modes such as chain-of-thought and inference-time scaling. However, reasoning also…
TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodput
Xiaoxuan Liu, Jongseok Park, Langxiang Hu +10
Large Language Model (LLM) serving systems batch concurrent user requests to achieve efficient serving. However, in real-world deployments, such inter-request parallelism from batc…
Optimizing LLM Queries in Relational Data Analytics Workloads
Shu Liu, Asim Biswal, Amog Kamsetty +8
Batch data analytics is a growing application for Large Language Models (LLMs). LLMs enable users to perform a wide range of natural language tasks, such as classification, entity…
Jenga: Effective Memory Management for Serving LLM with Heterogeneity
Chen Zhang, Kuntai Du, Shu Liu +10
Large language models (LLMs) are widely used but expensive to run, especially as inference workloads grow. To lower costs, maximizing the request batch size by managing GPU memory…
SkyStore: Cost-Optimized Object Storage Across Regions and Clouds
Shu Liu, Xiangxi Mo, Moshik Hershcovitch +11
Modern applications span multiple clouds to reduce costs, avoid vendor lock-in, and leverage low-availability resources in another cloud. However, standard object stores operate wi…
LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!
Dacheng Li, Shiyi Cao, Tyler Griggs +9
Large reasoning models (LRMs) tackle complex reasoning problems by following long chain-of-thoughts (Long CoT) that incorporate reflection, backtracking, and self-validation. Howev…