Human-Less LLM Serving: Quantifying the Human Tax on Throughput

Jianhui Lian, Li Chen, Dan Li +1

Every major LLM serving system is designed to meet TTFT and TPOT SLOs. These metrics capture latency as a human user perceives it, and the mechanisms built to satisfy them are now…