Cheaply Evaluating Inference Efficiency Metrics for Autoregressive Transformer APIs

Deepak Narayanan, Keshav Santhanam, Peter Henderson +3

Large language models (LLMs) power many state-of-the-art systems in natural language processing. However, these models are extremely computationally expensive, even at inference ti…