Efficient LLM Serving on Hybrid Real-time and Best-effort Requests

Wan Borui, Zhao Juntao, Jiang Chenyu +2

Recent breakthroughs in large Language Models (LLMs) have enabled various generative tasks on a single model. Real-world services (e.g., OpenAI's ChatGPT [27]) powered by an LLM of…