Scaling LLM Inference via Parallelism: Throughput-Latency Trade-off

Javad Mirzaei, Jeebak Mitra

Large Language Model (LLM) serving systems must balance latency and throughput while efficiently utilizing distributed GPU resources. This challenge arises because inference consis…