3 citations · 6 across the 12 of their papers we have counts for
13 papers
CascadeEP: Asynchronous Expert Execution for MoE Prefill under Attention Imbalance
Jin Qin, Tiancheng Hu, Shiyan Wang +12
Mixture-of-experts (MoE) serving commonly deploys data and expert parallelism (DEP): attention replicas run distinct request batches while routed experts are sharded across an expe…
RedKnot-MLA: Multi-Head Offline-Online Reuse for DeepSeek-V4 Long-Context Serving
Yang Liu, Zhaokai Luo, Huayi Jin +10
Multi-head latent attention (MLA) exposes many logical query heads through one packed latent KV stream. This representation is memory efficient, but it removes the physical per-hea…
StrataCL: Fabric-Native Communication Library for Production Supernodes
Tiancheng Hu, Jin Qin, Yuzheng Wang +14
Modern distributed AI workloads run across hundreds of accelerators, making communication a major bottleneck. Existing communication libraries remain largely buffer-centric because…
GEB-Bench: Abstract Structures Told in Many Voices
Tong Zhang, Zhiyuan Shi, Yun Peng +1
Analogy allows models to apply relational knowledge from one situation to another. Benchmarks of equivalent representations test different ways of expressing the same underlying pr…
When Does Recurrence Become an Algorithm? Convergence Selection in Weight-Tied Looped Transformers
Tong Zhang, Junhao Hu, Yun Peng +1
When does a weight-tied looped transformer -- one block applied T times -- implement an actual algorithm? We answer with four findings from controlled populations on group word pro…
Tessera: Unlocking Heterogeneous GPUs through Kernel-Granularity Disaggregation
Tiancheng Hu, Jin Qin, Zheng Wang +10
Disaggregation maps parts of an AI workload to different types of GPUs, offering a path to utilize modern heterogeneous GPU clusters. However, existing solutions operate at a coars…
Hummingbird: SLO-Oriented GPU Preemption at Microsecond-scale
Tiancheng Hu, Chenxi Wang, Ting Cao +9
Existing GPU-sharing techniques, including spatial and temporal sharing, aim to improve utilization but face challenges in simultaneously ensuring SLO adherence and maximizing effi…
Lil: Less is Less When Applying Post-Training Sparse-Attention Algorithms in Long-Decode Stage
Junhao Hu, Fangze Li, Mingtao Xu +9
Large language models (LLMs) demonstrate strong capabilities across a wide range of complex tasks and are increasingly deployed at scale, placing significant demands on inference e…
LLMigrate: Transforming "Lazy" Large Language Models into Efficient Source Code Migrators
Yuchen Liu, Junhao Hu, Yingdi Shan +4
Rewriting C code in Rust provides stronger memory safety, yet migrating large codebases such as the 32-million-line Linux kernel remains challenging. While rule-based translators (…
RaaS: Reasoning-Aware Attention Sparsity for Efficient LLM Reasoning
Junhao Hu, Wenrui Huang, Weidong Wang +6
Large Language Models (LLMs) have demonstrated strong capabilities across various domains, with recent advancements in challenging reasoning tasks such as mathematics and programmi…
DeepServe: Serverless Large Language Model Serving at Scale
Junhao Hu, Jiang Xu, Zhixia Liu +18
In this paper, we propose DEEPSERVE, a scalable and serverless AI platform designed to efficiently serve large language models (LLMs) at scale in cloud environments. DEEPSERVE addr…
EPIC: Efficient Position-Independent Caching for Serving Large Language Models
Junhao Hu, Wenrui Huang, Weidong Wang +7
Large Language Models (LLMs) show great capabilities in a wide range of applications, but serving them efficiently becomes increasingly challenging as requests (prompts) become mor…
MemServe: Context Caching for Disaggregated LLM Serving with Elastic Memory Pool
Cunchen Hu, Heyang Huang, Junhao Hu +8
Large language model (LLM) serving has transformed from stateless to stateful systems, utilizing techniques like context caching and disaggregated inference. These optimizations ex…