4 papers · 1 filter
Qrita: High-performance Top-k and Top-p using Pivot-based Truncation and Selection
Jongseok Park, Sunga Kim, Alvin Cheung +1
Despite their importance in model sampling, efficient implementation of Top-k and Top-p algorithms for large vocabularies remains a significant challenge. Existing approaches often…
Combee: Scaling Prompt Learning for Self-Improving Language Model Agents
Hanchen Li, Runyuan He, Qizheng Zhang +11
Recent advances in prompt learning allow large language model agents to acquire task-relevant knowledge from inference-time context without parameter changes. For example, existing…
TurboSpec: Closed-loop Speculation Control System for Optimizing LLM Serving Goodput
Xiaoxuan Liu, Jongseok Park, Langxiang Hu +10
Large Language Model (LLM) serving systems batch concurrent user requests to achieve efficient serving. However, in real-world deployments, such inter-request parallelism from batc…
Online Speculative Decoding
Xiaoxuan Liu, Lanxiang Hu, Peter Bailis +4
Speculative decoding is a pivotal technique to accelerate the inference of large language models (LLMs) by employing a smaller draft model to predict the target model's outputs. Ho…