Task Scheduling for Efficient Inference of Large Language Models on Single Moderate GPU Systems

Wenxiang Lin, Xinglin Pan, Shaohuai Shi +2

Large language models~(LLMs) are known for their high demand on computing resources and memory due to their substantial model size, which leads to inefficient inference on moderate…