Echo: Efficient Co-Scheduling of Hybrid Online-Offline Tasks for Large Language Model Serving

Zhibin Wang, Shipeng Li, Xue Li +7

Large language models have been widely deployed in various applications, encompassing both interactive online tasks and batched offline tasks. Given the burstiness and latency sens…