Showing cs.LGShow all
2 papers · 1 filter
cs.LG2026
IceCache: Memory-efficient KV-cache Management for Long-Sequence LLMs
Yuzhen Mao, Qitong Wang, Martin Ester +1
Key-Value (KV) cache plays a crucial role in accelerating inference in large language models (LLMs) by storing intermediate attention states and avoiding redundant computation duri…
cs.LG2024
IceFormer: Accelerated Inference with Long-Sequence Transformers on CPUs
Yuzhen Mao, Martin Ester, Ke Li
One limitation of existing Transformer-based models is that they cannot handle very long sequences as input since their self-attention operations exhibit quadratic time and space c…