Strata: Hierarchical Context Caching for Long Context Language Model Serving

Zhiqiang Xie, Ziyi Xu, Mark Zhao +5

Large Language Models (LLMs) with expanding context windows face significant performance hurdles. While caching key-value (KV) states is critical for avoiding redundant computation…