Etalon: Holistic Performance Evaluation Framework for LLM Inference Systems

Amey Agrawal, Anmol Agarwal, Nitin Kedia +5

Serving large language models (LLMs) in production can incur substantial costs, which has prompted recent advances in inference system optimizations. Today, these systems are evalu…