Auditing Prompt Caching in Language Model APIs

Chenchen Gu, Xiang Lisa Li, Rohith Kuditipudi +2

Prompt caching in large language models (LLMs) results in data-dependent timing variations: cached prompts are processed faster than non-cached prompts. These timing differences in…