SeaLLM: Service-Aware and Latency-Optimized Resource Sharing for Large Language Model Inference

Yihao Zhao, Jiadun Chen, Peng Sun +3

Large language models (LLMs) with different architectures and sizes have been developed. Serving each LLM with dedicated GPUs leads to resource waste and service inefficiency due t…