5 papers
Huawei Cloud Model-as-a-Service on the CloudMatrix384 SuperPod
Ao Xiao, Bangzheng He, Baoquan Zhang +125
Scaled-out MoE LLMs and scaled-up SuperPods create new systems challenges for production Model-as-a-Service (MaaS), requiring disaggregation, low-latency communication, and decentr…
ReviveMoE: Fast Recovery for Hardware Failures in Large-Scale MoE LLM Inference Deployments
Haley Li, Xinglu Wang, Cong Feng +12
As LLM deployments scale over more hardware, the probability of a single failure in a system increases significantly, and cloud operators must consider robust countermeasures to ha…
LLMPrism: Black-box Performance Diagnosis for Production LLM Training Platforms
Zhihan Jiang, Rui Ren, Guangba Yu +8
Large Language Models (LLMs) have brought about revolutionary changes in diverse fields, rendering LLM training of utmost importance for modern enterprises. To meet this demand, mu…
L4: Diagnosing Large-scale LLM Training Failures via Automated Log Analysis
Zhihan Jiang, Junjie Huang, Zhuangbin Chen +6
As Large Language Models (LLMs) show their capabilities across various applications, training customized LLMs has become essential for modern enterprises. However, due to the compl…
Identifying Performance Issues in Cloud Service Systems Based on Relational-Temporal Features
Wenwei Gu, Jinyang Liu, Zhuangbin Chen +7
Cloud systems are susceptible to performance issues, which may cause service-level agreement violations and financial losses. In current practice, crucial metrics are monitored per…