2 citations · 2 across the 13 of their papers we have counts for
15 papers
Herschel: Continuous Optimization of Production LLM Inference through On-Demand Profiling
Luping Wang, Weigao Chen, Yifei Wu +11
Model-as-a-service platforms call for continuous optimization as complex serving conditions expose inefficiencies missed before deployment. Detailed always-on profiling can incur s…
Capture the lifecycle: KV Cache management in ReAct Agents with KVTether
Kaihua Fu, Yukun Zhou, Chaokun Chang +6
Efficient serving of long-context reasoning-and-acting (ReAct) agents relies on KV cache reuse to reduce large language model (LLM) prefill latency and monetary cost. However, a se…
Weave: Fine-Grained Dynamic SM Scheduling in an MoE Megakernel for Compute-Communication Overlap
Ziyu Huang, Yangjie Zhou, Chenhao Zhu +15
Mixture-of-Experts (MoE) inference under expert parallelism (EP) turns each MoE layer into a distributed computation with costly dispatch and combine communication. State-of-the-ar…
Are LLM-Generated GPU Kernels Production-Ready? A Trace-Driven Benchmark and Optimization Agent
Lingyun Yang, Yuxiao Wang, Shenghao Liang +8
Existing GPU kernel generation benchmarks draw problems from synthetic or curated sources that diverge from deployed workloads. We present Atrex-Bench, a benchmark whose 30 operato…
Ling and Ring 2.6 Technical Report: Efficient and Instant Agentic Intelligence at Trillion-Parameter Scale
Ang Li, Ben Liu, Bin Han +215
Efficient and scalable agentic intelligence requires models that can deliver both low-latency responses and strong reasoning capabilities while remaining practical to train, serve,…
WaveTune: Wave-aware Bilinear Modeling for Efficient GPU Kernel Auto-tuning
Kaixuan Zhang, Chutong Ding, Shiyou Qian +6
The rapid adoption of Large Language Models (LLMs) has made GPU inference efficiency an increasingly critical system concern. The runtime of LLM workloads is largely dominated by t…
LegoDiffusion: Micro-Serving Text-to-Image Diffusion Workflows
Lingyun Yang, Suyi Li, Tianyu Feng +10
Text-to-image generation executes a diffusion workflow comprising multiple models centered on a base diffusion model. Existing serving systems treat each workflow as an opaque mono…
Dissecting Outlier Dynamics in LLM NVFP4 Pretraining
Peijie Dong, Ruibo Fan, Yuechen Tao +11
Training large language models using 4-bit arithmetic enhances throughput and memory efficiency. Yet, the limited dynamic range of FP4 increases sensitivity to outliers. While NVFP…
PipeWeave: Synergizing Analytical and Learning Models for Unified GPU Performance Prediction
Kaixuan Zhang, Yunfan Cui, Shuhao Zhang +8
The rapid expansion of Transformer-based large language models has dramatically increased the need for high-performance GPUs. As a result, there is growing demand for fast, accurat…
GFS: A Preemption-aware Scheduling Framework for GPU Clusters with Predictive Spot Instance Management
Jiaang Duan, Shenglin Xu, Shiyou Qian +15
The surge in large language models (LLMs) has fundamentally reshaped the landscape of GPU usage patterns, creating an urgent need for more efficient management strategies. While cl…
Huawei Cloud Model-as-a-Service on the CloudMatrix384 SuperPod
Ao Xiao, Bangzheng He, Baoquan Zhang +125
Scaled-out MoE LLMs and scaled-up SuperPods create new systems challenges for production Model-as-a-Service (MaaS), requiring disaggregation, low-latency communication, and decentr…
InstGenIE: Generative Image Editing Made Efficient with Mask-aware Caching and Scheduling
Xiaoxiao Jiang, Suyi Li, Lingyun Yang +12
Generative image editing using diffusion models has become a prevalent application in today's AI cloud services. In production environments, image editing typically involves a mask…
Adaptra: Straggler-Resilient Hybrid-Parallel Training with Pipeline Adaptation
Tianyuan Wu, Lunxi Cao, Hanfeng Lu +8
Training large Deep Neural Network (DNN) models at scale often encounters straggler issues, mostly in communications due to network congestion, RNIC/switch defects, or topological…
FALCON: Pinpointing and Mitigating Stragglers for Large-Scale Hybrid-Parallel Training
Tianyuan Wu, Wei Wang, Yinghao Yu +7
Fail-slows, or stragglers, are common but largely unheeded problems in large-scale hybrid-parallel training that spans thousands of GPU servers and runs for weeks to months. Yet, t…
SwiftDiffusion: Efficient Diffusion Model Serving with Add-on Modules
Suyi Li, Lingyun Yang, Xiaoxiao Jiang +12
Text-to-image (T2I) generation using diffusion models has become a blockbuster service in today's AI cloud. A production T2I service typically involves a serving workflow where a b…