1 citations · 1 across the 7 of their papers we have counts for
10 papers
BELT : BEhavior-Aligned Low-Rank KV Cache Compression with Learned Top-k Rank Allocation
Yaoyang Liu, Shuwei Shen, Jincheng Tang +5
Long-context inference makes the key-value (KV) cache a major memory bottleneck for large language models. We propose BELT, a low-rank KV cache compression framework. Its key-side…
SpecFit: Not All Draft Tokens Cost the Same in Offloaded MoE Inference
Hongliang Cao, Xiuyu Li, Siqi Wang +3
Mixture-of-experts (MoE) models activate only a few experts per token, but when their expert weights are offloaded to host memory, host-to-device (H2D) expert transfers enter the c…
SPADE: An Input-Adaptive Sparse Attention Engine for Fast Video Diffusion Models Inference
Shanghao Liu, Renze Chen, Size Zheng +4
Video diffusion transformers (vDiTs) generate high quality but pay quadratic self-attention cost, making inference prohibitive at video-token scales. The challenge is input-adaptiv…
OOCO: Latency-disaggregated Architecture for Online-Offline Co-locate LLM Serving
Siyu Wu, Zihan Tang, Yuting Zeng +5
Large Language Models (LLMs) are increasingly deployed in both latency-sensitive online services and cost-sensitive offline workloads. Co-locating these workloads on shared serving…
xLLM Technical Report
Tongxuan Liu, Tao Peng, Peijun Yang +50
We introduce xLLM, an intelligent and efficient Large Language Model (LLM) inference framework designed for high-performance, large-scale enterprise-grade serving, with deep optimi…
Pre: Enabling Deterministic Pushdown Automata for Faster Structured LLM Generation
Junyi Chen, Shihao Bai, Zaijun Wang +7
Extensive LLM applications demand efficient structured generations, particularly for LR(1) grammars, to produce outputs in specified formats (e.g., JSON). Existing methods primaril…
HydraInfer: Hybrid Disaggregated Scheduling for Multimodal Large Language Model Serving
Xianzhe Dong, Tongxuan Liu, Yuting Zeng +7
Multimodal Large Language Models (MLLMs) have been rapidly advancing, enabling cross-modal understanding and generation, and propelling artificial intelligence towards artificial g…
Arrow: Adaptive Scheduling Mechanisms for Disaggregated LLM Inference Architecture
Yu Wu, Tongxuan Liu, Yuting Zeng +6
Existing large language model (LLM) serving systems typically employ Prefill-Decode disaggregated architecture to prevent computational interference between the prefill and decode…
Logic-of-Thought: Injecting Logic into Contexts for Full Reasoning in Large Language Models
Tongxuan Liu, Wenjiang Xu, Weizhe Huang +5
Large Language Models (LLMs) have demonstrated remarkable capabilities across various tasks but their performance in complex logical reasoning tasks remains unsatisfactory. Althoug…
GroupDebate: Enhancing the Efficiency of Multi-Agent Debate Using Group Discussion
Tongxuan Liu, Xingyu Wang, Weizhe Huang +5
In recent years, Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse NLP tasks. Extensive research has explored how to enhance the logical reasoni…