3 papers
cs.DC2025
Staggered Batch Scheduling: Co-optimizing Time-to-First-Token and Throughput for High-Efficiency LLM Inference
Jian Tian, Shuailong Li, Yang Cao +8
The evolution of Large Language Model (LLM) serving towards complex, distributed architectures--specifically the P/D-separated, large-scale DP+EP paradigm--introduces distinct sche…
cs.CL2025
Traversal Verification for Speculative Tree Decoding
Yepeng Weng, Qiao Hu, Xujie Chen +5
Speculative decoding is a promising approach for accelerating large language models. The primary idea is to use a lightweight draft model to speculate the output of the target mode…
cs.CL2025
CORAL: Learning Consistent Representations across Multi-step Training with Lighter Speculative Drafter
Yepeng Weng, Dianwen Mei, Huishi Qiu +4
Speculative decoding is a powerful technique that accelerates Large Language Model (LLM) inference by leveraging a lightweight speculative draft model. However, existing designs su…