AugServe: Adaptive Request Scheduling for Augmented Large Language Model Inference Serving

Ying Wang, Zhen Jin, Jiexiong Xu +3

As augmented large language models (LLMs) with external tools become increasingly popular in web applications, improving augmented LLM inference serving efficiency and optimizing s…