OmniInfer: System-Wide Acceleration Techniques for Optimizing LLM Serving Throughput and Latency

Jun Wang, Yunxiang Yao, Wenwei Kuang +11

Large Language Models drive a wide range of modern AI applications but impose substantial challenges on large-scale serving systems due to intensive computation, strict latency con…