Argus: Token Aware Distributed LLM Inference Optimization

Panlong Wu, Yifei Zhong, Danyang Chen +2

Large Language Models (LLMs) are rapidly being integrated into real-world applications, yet their autoregressive architectures introduce significant inference time variability, esp…