LatencyPrism: Online Non-intrusive Latency Sculpting for SLO-Guaranteed LLM Inference

Yin Du, Jiayi Ren, Xiayu Sun +4

LLM inference latency critically determines user experience and operational costs, directly impacting throughput under SLO constraints. Even brief latency spikes degrade service qu…