5 papers
ProServe: Unified Multi-Priority Request Scheduling for LLM Serving
Weizhe Huang, Tao Peng, Tongxuan Liu +4
The widespread deployment of large language models (LLMs) for interactive applications necessitates serving systems that can handle thousands of concurrent requests with diverse Se…
xLLM Technical Report
Tongxuan Liu, Tao Peng, Peijun Yang +50
We introduce xLLM, an intelligent and efficient Large Language Model (LLM) inference framework designed for high-performance, large-scale enterprise-grade serving, with deep optimi…
IFDNS: An Iterative Feedback-Driven Neuro-Symbolic Method for Faithful Logical Reasoning
Xiaoheng Wang, Tongxuan Liu, Zi Gong +5
Large language models (LLMs) have demonstrated impressive capabilities across a wide range of reasoning tasks, including logical and mathematical problem-solving. While prompt-base…
HydraInfer: Hybrid Disaggregated Scheduling for Multimodal Large Language Model Serving
Xianzhe Dong, Tongxuan Liu, Yuting Zeng +7
Multimodal Large Language Models (MLLMs) have been rapidly advancing, enabling cross-modal understanding and generation, and propelling artificial intelligence towards artificial g…
Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecture
Yu Wu, Tongxuan Liu, Yuting Zeng +6
Existing large language model (LLM) serving systems typically employ Prefill-Decode disaggregated architecture to prevent computational interference between the prefill and decode…