4 papers
The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs
Piotr Nawrot, Robert Li, Renjie Huang +3
Sparse attention offers a promising strategy to extend long-context capabilities in Transformer LLMs, yet its efficiency-accuracy trade-offs remain unclear due to the lack of compr…
Fast and Expressive Multi-Byte Prediction with Probabilistic Circuits
Andreas Grivas, Lorenzo Loconte, Emile van Krieken +6
Multi-token prediction (MTP) is a prominent strategy to significantly speed up generation in large language models (LLMs), especially in byte-level LLMs, which are tokeniser-free b…
Inference-Time Hyper-Scaling with KV Cache Compression
Adrian ÅaÅcucki, Konrad Staniszewski, Piotr Nawrot +1
Inference-time scaling trades efficiency for increased reasoning accuracy by generating longer or more parallel sequences. However, in Transformer LLMs, generation cost is bottlene…
Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference
Piotr Nawrot, Adrian ÅaÅcucki, Marcin Chochowski +2
Transformers have emerged as the backbone of large language models (LLMs). However, generation remains inefficient due to the need to store in memory a cache of key-value represent…