A Generative Caching System for Large Language Models

Arun Iyengar, Ashish Kundu, Ramana Kompella +1

Caching has the potential to be of significant benefit for accessing large language models (LLMs) due to their high latencies which typically range from a small number of seconds t…