OmniKVQuant: KV Cache Quantization for Omni-LLMs

Suho Yoo, Hyunjong Ok, Jongmin Choi +2

As Omni-modal large language models (Omni-LLMs) take in audio, video and text together, their KV cache memory cost grows. KV cache quantization is the de facto approach in text-onl…