Enhancing reliability in AI inference services: An empirical study on real production incidents

Bhala Ranganathan, Mickey Zhang, Kai Wu

Hyperscale large language model (LLM) inference places extraordinary demands on cloud systems, where even brief failures can translate into significant user and business impact. To…