Showing cs.LGShow all
3 papers · 1 filter
cs.LG2025
The Pitfalls of KV Cache Compression
Alex Chen, Renato Geh, Aditya Grover +2
KV cache compression promises increased throughput and efficiency with negligible loss in performance. While the gains in throughput are indisputable and recent literature has inde…
cs.LG2025
Enabling Autoregressive Models to Fill In Masked Tokens
Daniel Israel, Aditya Grover, Guy Van den Broeck
Historically, LLMs have been trained using either autoregressive (AR) or masked language modeling (MLM) objectives, with AR models gaining dominance in recent years. However, AR mo…
cs.LG2024
Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models
Siyan Zhao, Daniel Israel, Guy Van den Broeck +1
During inference for transformer-based large language models (LLM), prefilling is the computation of the key-value (KV) cache for input tokens in the prompt prior to autoregressive…