Meta-Metrics and Best Practices for System-Level Inference Performance Benchmarking

Shweta Salaria, Zhuoran Liu, Nelson Mimura Gonzalez

Benchmarking inference performance (speed) of Foundation Models such as Large Language Models (LLM) involves navigating a vast experimental landscape to understand the complex inte…