Inference performance evaluation for LLMs on edge devices with a novel benchmarking framework and metric

Hao Chen, Cong Tian, Zixuan He +3

With the significant success achieved by large language models (LLMs) like LLaMA, edge computing-based LLM inference services for mobile and PC are in high demand for data privacy.…