10 papers · 1 filter
LeanSparse: Memory-Efficient Serving for Native Sparse-Attention LLMs
Zaifeng Pan, Chris Wu, Xinwei Qiang +3
Sparse attention limits the history read by each query, but serving long contexts still requires substantial GPU memory. Existing KV offloading systems retain the full history of i…
Weights That Stay: On-Chip Megakernels for Serial Heads in Speculative Decoding
Xinwei Qiang, Ziyi Xu, Zaifeng Pan +2
Parallel speculative drafters use a lightweight serial head to recover token dependencies within a block generated in parallel. Each head step updates only a small state, yet proje…
Understanding GPU Megakernels: Why Residency Can Mislead
Xinwei Qiang, Ziyi Xu, Chang Chen +4
Megakernels let one persistent GPU program prepare dependent operators early and execute ready tasks together. We study the benefits and costs of these capabilities by reconstructi…
HiState: Hierarchical Prefix Cache Management for Hybrid LLM Serving in Agentic Workloads
Zaifeng Pan, Lucas Wang, Xinwei Qiang +5
Hybrid large language models combine full-attention and recurrent linear layers to reduce inference costs. Prefix caching can further reduce redundant computation across requests,…
Zepp: Accelerating Distributed MoE Serving under Relaxed Balance Constraints
Chang Chen, Andrew Yang, Tiancheng Chen +5
As Mixture-of-Experts (MoE) models continue to scale, serving them increasingly relies on expert parallelism (EP) across a growing number of devices. Yet skewed expert workloads cr…
AgentReplay: Token-Wise Trace Replay Is Essential for Fair Serving System Performance Benchmarking
Zaifeng Pan, Michael Wang, Chris Wu +4
LLM-based agents execute multi-turn workflows with interleaved model inference and tool calls, making efficient serving increasingly important. However, evaluating serving optimiza…
RR-Evict: Fine-Grained Prefix Cache Eviction beyond LRU for Agentic LLM Serving
Zaifeng Pan, Chris Wu, Zhengding Hu +3
LLM-based agents execute long-horizon tasks through repeated model calls interleaved with tool execution and user interaction. As each call extends the history accumulated in previ…
Argus: Orchestrating Cross-Layer GPU Performance Measurements around Semantic Regions
Jianzhu Yao, Yue Guan, Srivatsan Ramesh +11
GPU developers and automated optimizers need performance evidence for semantic code regions--such as neural-network operator implementations and pipeline stages--but this evidence…
GF-DiT: Scheduling Parallelism for Diffusion Transformer Serving
Xinwei Qiang, Yifan Hu, Shixuan Sun +6
Diffusion Transformers (DiTs) have become the dominant architecture for image and video generation, creating growing demand for efficient DiT serving. Existing systems assign each…
Syncopate: Efficient Multi-GPU AI Kernels via Automatic Chunk-Centric Compute-Communication Overlap
Xinwei Qiang, Yue Guan, Zhengding Hu +3
Communication has become a first-order bottleneck in large-scale GPU workloads, and existing distributed compilers address it mainly by overlapping whole compute and communication…