ExeGPT: Constraint-Aware Resource Scheduling for LLM Inference

Hyungjun Oh, Kihong Kim, Jaemin Kim +4

This paper presents ExeGPT, a distributed system designed for constraint-aware LLM inference. ExeGPT finds and runs with an optimal execution schedule to maximize inference through…