1 citations · 3 across the 12 of their papers we have counts for
12 papers
Breaking the Reward Barrier: Accelerating Tree-of-Thought Reasoning via Speculative Exploration
Shuzhang Zhong, Haochen Huang, Shengxuan Qiu +3
Tree-of-Thought (ToT) reasoning structures Large Language Model (LLM) inference as a tree-based search, demonstrating strong potential for solving complex mathematical and programm…
DualMap: Enabling Both Cache Affinity and Load Balancing for Distributed LLM Serving
Ying Yuan, Pengfei Zuo, Bo Wang +3
In LLM serving, reusing the KV cache of prompts across requests is critical for reducing TTFT and serving costs. Cache-affinity scheduling, which co-locates requests with the same…
HyPER: Bridging Exploration and Exploitation for Scalable LLM Reasoning with Hypothesis Path Expansion and Reduction
Shengxuan Qiu, Haochen Huang, Shuzhang Zhong +2
Scaling test-time compute with multi-path chain-of-thought improves reasoning accuracy, but its effectiveness depends critically on the exploration-exploitation trade-off. Existing…
RelayGR: Scaling Long-Sequence Generative Recommendation via Cross-Stage Relay-Race Inference
Jiarui Wang, Huichao Chai, Yuanhang Zhang +38
Real-time recommender systems execute multi-stage cascades (retrieval, pre-processing, fine-grained ranking) under strict tail-latency SLOs, leaving only tens of milliseconds for r…
Lotus: Optimizing Disaggregated Transactions with Disaggregated Locks
Zhisheng Hu, Pengfei Zuo, Junliang Hu +3
Disaggregated memory (DM) separates compute and memory resources, allowing flexible scaling to achieve high resource utilization. To ensure atomic and consistent data access on DM,…
SparseServe: Unlocking Parallelism for Dynamic Sparse Attention in Long-Context LLM Serving
Qihui Zhou, Peiqi Yin, Pengfei Zuo +1
Serving long-context LLMs is costly because attention computation grows linearly with context length. Dynamic sparse attention algorithms (DSAs) mitigate this by attending only to…
Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving
Chao Wang, Pengfei Zuo, Zhangyu Chen +3
An ongoing debate considers whether prefill-decode (PD) aggregation or disaggregation is superior for serving large language models (LLMs). This has driven optimizations for both a…
Serving Large Language Models on Huawei CloudMatrix384
Pengfei Zuo, Huimin Lin, Junbo Deng +43
The rapid evolution of large language models (LLMs), driven by growing parameter scales, adoption of mixture-of-experts (MoE) architectures, and expanding context lengths, imposes…
Injecting Adrenaline into LLM Serving: Boosting Resource Utilization and Throughput via Attention Disaggregation
Yunkai Liang, Zhangyu Chen, Pengfei Zuo +3
In large language model (LLM) serving systems, executing each request consists of two phases: the compute-intensive prefill phase and the memory-intensive decoding phase. To preven…
Progressive Sparse Attention: Algorithm and System Co-design for Efficient Attention in LLM Serving
Qihui Zhou, Peiqi Yin, Pengfei Zuo +1
Processing long contexts has become a critical capability for modern large language models (LLMs). However, serving long-context LLMs comes with significant inference costs due to…
AdaSkip: Adaptive Sublayer Skipping for Accelerating Long-Context LLM Inference
Zhuomin He, Yizhen Yao, Pengfei Zuo +4
Long-context large language models (LLMs) inference is increasingly critical, motivating a number of studies devoted to alleviating the substantial storage and computational costs…
Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention
Bin Gao, Zhuomin He, Puru Sharma +6
Interacting with humans through multi-turn conversations is a fundamental feature of large language models (LLMs). However, existing LLM serving engines executing multi-turn conver…