8 papers
VenusRL: A Fully Disaggregated Agentic RL System with Priority Scheduling and Scalable Interaction
Mingjun Zhang, Yucheng Li, Menghao Zhang +15
Agentic Reinforcement Learning (RL) trains LLM agents through multi-turn interactions with external tool environments. Its multi-turn nature exposes two system-level bottlenecks un…
Efficient Agentic LLM Serving over SSD-based Sparse KV Storage
Wenhao He, Ping Zhang, Xiaohe Hu +5
Agentic sessions driven by Large language models (LLMs) often alternate between model inference and tool use, accumulating long histories across successive rounds. Serving these se…
Demystifying the Design Space and Best Practices for Heterogeneous LLM Inference and Serving
Zhixin Wang, Zhengbo Wang, Fangcheng Fu +15
Heterogeneous prefill-decode (PD) inference is now in production: prefill on cost-efficient or supply-available accelerators, decode on bandwidth-strong ones, and KV state crossing…
PlexRL: Cluster-Level Orchestration of Serviceized LLM Execution for RLVR
Yiqi Zhang, Fangzheng Jiao, Tian Tang +13
Reinforcement learning with verifiable rewards (RLVR) has recently unlocked strong reasoning capabilities in large language models (LLMs), triggering rapid exploration of new algor…
EPIC: Abstraction and Polymorphism of In-Network Collectives on Ethernet
Yitao Yuan, Jianglong Nie, Tianyu Bai +28
In-Network Collective (INC) acceleration holds immense potential for optimizing AI training and inference; however, its cross-layer nature has historically hindered investment and…
Learning to Wait: Synchronizing Agents with the Physical World
Yifei She, Ping Zhang, He Liu +6
Real-world agentic tasks, unlike synchronous Markov Decision Processes (MDPs), often involve non-blocking actions with variable latencies, creating a fundamental \textit{Temporal G…
An Efficient, Reliable and Observable Collective Communication Library in Large-scale GPU Training Clusters
Mingjun Zhang, Xiaohe Hu, Menghao Zhang +21
Large-scale LLM training requires collective communication libraries to exchange data among distributed GPUs. As a company dedicated to building and operating large-scale GPU train…
Expert-as-a-Service: Towards Efficient, Scalable, and Robust Large-scale MoE Serving
Ziming Liu, Boyu Tian, Guoteng Wang +15
Mixture-of-Experts (MoE) models challenge serving infrastructures with dynamic, sparse expert utilization, causing instability on conventional systems designed for dense architectu…