4 papers
Bole: Efficient Tree Speculation for Hybrid-Attention Language Models
Li Wang, Yi Su, Xiabao Wu +9
Hybrid-attention large language models combine full attention with recurrent linear attention to reduce long-context inference costs, yet their autoregressive decoding remains memo…
Decoupled Attention Fusion: Accelerating RAG with Efficient KV Cache Reuse
Xiabao Wu, Wentao Liu, Yongchao Liu +1
Retrieval-Augmented Generation (RAG) effectively mitigates hallucinations in Large Language Models (LLMs) but suffers from prohibitive Time-To-First-Token (TTFT) latency in long-co…
Scaling Graph Chain-of-Thought Reasoning: A Multi-Agent Framework with Efficient LLM Serving
Chengying Huan, Ziheng Meng, Yongchao Liu +11
Graph Chain-of-Thought (Graph-CoT) enables large language models (LLMs) to perform step-by-step reasoning over graph-structured knowledge, but existing pipelines suffer from low ac…
Distributed Graph Neural Network Inference With Just-In-Time Compilation For Industry-Scale Graphs
Xiabao Wu, Yongchao Liu, Wei Qin +1
Graph neural networks (GNNs) have delivered remarkable results in various fields. However, the rapid increase in the scale of graph data has introduced significant performance bott…