Load Balancing with Evolving Workloads in Large-Scale Language Model Serving

Zixi Chen, Tianci Bu, Chendong Song +3

Serving large language models requires assigning requests whose workloads grow during processing and whose completion times are unknown. With a synchronization barrier, lightly loa…