Transferable Latency Prediction for Fast LLM Screening on Heterogeneous Edge Devices

Xiaolong Tu, Vinod K. Mishra, Venkat R. Dasari +2

Accurate latency prediction is critical for deploying large language models (LLMs) on heterogeneous edge devices, where inference latency is affected by model architecture, prompt…