7 citations · 25 across the 24 of their papers we have counts for
4 papers · 1 filter
WUSH-KV: KV Cache Quantization with Data-Adaptive Transforms
Jiale Chen, Vage Egiazarian, Eldar Kurtić +2
KV cache memory and bandwidth costs grow with context length and batch size, which limits efficient long-context inference. To address this bottleneck, we introduce WUSH-KV for low…
H-Spec: Parallel Speculative Decoding Without a Drafter-Side KV Cache
Weifan Jiang, Krishna Teja Chitty-Venkata, Megan Flynn +6
Speculative decoding losslessly accelerates large language model inference by having a lightweight draft model predict future tokens for verification by the target model. Recent bl…
GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling
Alireza Dadgarnia, Soroush Tabesh, Mahdi Nikdan +4
Quantization has become a standard tool for efficient LLM deployment, especially for local inference, where models are now routinely served at 2-3 bits per parameter. The state of…
Statistically-Lossless Quantization of Large Language Models
Michael Helcig, Eldar Kurtic, Dan Alistarh
Model quantization has become essential for efficient large language model deployment, yet existing approaches present clear trade-offs: methods such as GPTQ and AWQ achieve practi…