FastSwitch: Optimizing Context Switching Efficiency in Fairness-aware Large Language Model Serving

Ao Shen, Zhiyao Li, Mingyu Gao

Serving numerous users and requests concurrently requires good fairness in Large Language Models (LLMs) serving system. This ensures that, at the same cost, the system can meet the…