17 citations · 26 across the 6 of their papers we have counts for
4 papers · 1 filter
The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs
Piotr Nawrot, Robert Li, Renjie Huang +3
Sparse attention offers a promising strategy to extend long-context capabilities in Transformer LLMs, yet its efficiency-accuracy trade-offs remain unclear due to the lack of compr…
Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference
Piotr Nawrot, Adrian Łańcucki, Marcin Chochowski +2
Transformers have emerged as the backbone of large language models (LLMs). However, generation remains inefficient due to the need to store in memory a cache of key-value represent…
nanoT5: A PyTorch Framework for Pre-training and Fine-tuning T5-style Models with Limited Resources
Piotr Nawrot
State-of-the-art language models like T5 have revolutionized the NLP landscape, but their computational demands hinder a large portion of the research community. To address this ch…
Efficient Transformers with Dynamic Token Pooling
Piotr Nawrot, Jan Chorowski, Adrian Łańcucki +1
Transformers achieve unrivalled performance in modelling language, but remain inefficient in terms of memory and time complexity. A possible remedy is to reduce the sequence length…