RT-LM: Uncertainty-Aware Resource Management for Real-Time Inference of Language Models

Yufei Li, Zexin Li, Wei Yang +1

Recent advancements in language models (LMs) have gained substantial attentions on their capability to generate human-like responses. Though exhibiting a promising future for vario…