9 papers
MARLaaS: Multi-Tenant Asynchronous Reinforcement Learning as a Service
Timothy Tin Long Yu, Gursimran Singh, Ge Shi +3
Reinforcement Learning from Verifiable Rewards (RLVR) has significantly improved the reasoning capabilities of large language models (LLMs), particularly in multi-turn agentic sett…
HFX: Joint Design of Algorithms and Systems for Multi-SLO Serving and Fast Scaling
Zahra Yousefijamarani, Xinglu Wang, Qian Wang +12
Large language model (LLM) serving faces the dual challenge of meeting strict user-specific service-level objectives (SLOs) while minimizing computational cost under dynamic, multi…
Huawei Cloud Model-as-a-Service on the CloudMatrix384 SuperPod
Ao Xiao, Bangzheng He, Baoquan Zhang +125
Scaled-out MoE LLMs and scaled-up SuperPods create new systems challenges for production Model-as-a-Service (MaaS), requiring disaggregation, low-latency communication, and decentr…
ReviveMoE: Fast Recovery for Hardware Failures in Large-Scale MoE LLM Inference Deployments
Haley Li, Xinglu Wang, Cong Feng +12
As LLM deployments scale over more hardware, the probability of a single failure in a system increases significantly, and cloud operators must consider robust countermeasures to ha…
DECKBench: Benchmarking Multi-Agent Frameworks for Academic Slide Generation and Editing
Daesik Jang, Morgan Lindsay Heisler, Linzi Xing +5
Automatically generating and iteratively editing academic slide decks requires more than document summarization. It demands faithful content selection, coherent slide organization,…
MEPIC: Memory Efficient Position Independent Caching for LLM Serving
Qian Wang, Zahra Yousefijamarani, Morgan Lindsay Heisler +8
Modern LLM applications such as deep-research assistants, coding agents, and Retrieval-Augmented Generation (RAG) systems, repeatedly process long prompt histories containing share…