Conveyor: Efficient Tool-aware LLM Serving with Tool Partial Execution

Yechen Xu, Xinhao Kong, Tingjun Chen +1

The complexity of large language model (LLM) serving workloads has substantially increased due to the integration with external tool invocations, such as ChatGPT plugins. In this p…