9 papers
SpecGen: Accelerating Agentic Kernel Optimization with Speculative Generation
Jihu Guo, Sitian Lu, Tenghui Ma +4
Agentic kernel optimization automates manual GPU kernel tuning via iterative generation, validation, and profiling with reasoning LLMs, casting the optimization task as feedback-gu…
DisagFusion: Asynchronous Pipeline Parallelism and Elastic Scheduling for Disaggregated Diffusion Serving
Hantian Zha, Teng Ma, Yang Yong +7
Diffusion-based generation is increasingly powering production content pipelines; however, deploying these models at scale remains a significant challenge. Model weights frequently…
ROSE: Rollout On Serving GPUs via Cooperative Elasticity for Agentic RL
Wei Gao, Yuheng Zhao, Dilxat Muhtar +13
Agentic reinforcement learning (RL) is reshaping LLM post-training, but end-to-end training time is dominated by compute-intensive, multi-turn rollouts whose resource demand varies…
ResiHP: Taming LLM Training Failures with Dynamic Hybrid Parallelism
Tenghui Ma, Jihu Guo, Wei Gao +4
Hybrid parallelism underpins large-scale LLM training across tens of thousands of GPUs. At such scale, hardware failures on individual devices lead to performance skew across devic…
Pooling Engram Conditional Memory in Large Language Models using CXL
Ruiyang Ma, Teng Ma, Zhiyuan Su +11
Engram conditional memory has emerged as a promising component for LLMs by decoupling static knowledge lookup from dynamic computation. Since Engram exhibits sparse access patterns…
Adaptive Multi-Objective Tiered Storage Configuration for KV Cache in LLM Service
Xianzhe Zheng, Zhengheng Wang, Ruiyan Ma +17
The memory-for-computation paradigm of KV caching is essential for accelerating large language model (LLM) inference service, but limited GPU high-bandwidth memory (HBM) capacity m…
TokenCake: A KV-Cache-centric Serving Framework for LLM-based Multi-Agent Applications
Zhuohang Bian, Feiyang Wu, Zhuoran Li +2
Large Language Models (LLMs) are increasingly deployed in complex multi-agent applications that rely on external function calls. This workload creates severe performance challenges…
OctoPipe: Reducing Pipeline Bubbles for Heterogeneous Models via Co-Optimizing Partitioning, Placement, and Scheduling
Jihu Guo, Tenghui Ma, Wei Gao +6
Pipeline parallelism is widely used to train large language models (LLMs). However, increasing heterogeneity in model architectures exacerbates pipeline bubbles, thereby reducing t…
TokenSim: Enabling Hardware and Software Exploration for Large Language Model Inference Systems
Feiyang Wu, Zhuohang Bian, Guoyang Duan +6
The increasing demand for large language model (LLM) serving has necessitated significant advancements in the optimization and profiling of LLM inference systems. As these models b…