Teola: Towards End-to-End Optimization of LLM-based Applications

Xin Tan, Yimin Jiang, Yitao Yang +1

Large language model (LLM)-based applications consist of both LLM and non-LLM components, each contributing to the end-to-end latency. Despite great efforts to optimize LLM inferen…