Efficient Prompt Caching via Embedding Similarity

Hanlin Zhu, Banghua Zhu, Jiantao Jiao

Large language models (LLMs) have achieved huge success in numerous natural language process (NLP) tasks. However, it faces the challenge of significant resource consumption during…