MILLION: Mastering Long-Context LLM Inference Via Outlier-Immunized KV Product Quantization

Zongwu Wang, Peng Xu, Fangxin Liu +7

Large language models (LLMs) are increasingly utilized for complex tasks requiring longer context lengths, with some models supporting up to 128K or 1M tokens. This trend, however,…