SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs

Shibo Jie, Yehui Tang, Kai Han +2

Transformer-based large language models (LLMs) have already achieved remarkable results on long-text tasks, but the limited GPU memory (VRAM) resources struggle to accommodate the…