Teola: Towards End-to-End Optimization of LLM-based Applications

Abstract

Large language model (LLM)-based applications consist of both LLM and non-LLM components, each contributing to the end-to-end latency. Despite great efforts to optimize LLM inference, end-to-end workflow optimization has been overlooked. Existing frameworks employ coarse-grained orchestration with task modules, which confines optimizations to within each module and yields suboptimal scheduling decisions. We propose fine-grained end-to-end orchestration, which utilizes task primitives as the basic units and represents each query's workflow as a primitive-level dataflow graph. This explicitly exposes a much larger design space, enables optimizations in parallelization and pipelining across primitives of different modules, and enhances scheduling to improve application-level performance. We build Teola, a novel orchestration framework for LLM-based applications that implements this scheme. Comprehensive experiments show that Teola can achieve up to 2.09x speedup over existing systems across various popular LLM applications. The code is available at https://github.com/NetX-lab/Ayo.

References in corpus (8)

• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face
• SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
• SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification
• Clipper: A Low-Latency Online Prediction Serving System
• AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving
• FlashDecoding++: Faster Large Language Model Inference on GPUs
• Parrot: Efficient Serving of LLM-based Applications with Semantic Variable