2 papers
cs.LG2025
Slim attention: cut your context memory in half without loss -- K-cache is all you need for MHA
Nils Graef, Andrew Wasielewski, Kayvan Zahiri
Slim attention shrinks the context memory size by 2x for transformer models with MHA (multi-head attention), which can speed up inference by up to 2x for large context windows. Sli…
cs.LG2024
FlashNorm: Fast Normalization for Transformers
Nils Graef, Filip Makraduli, Andrew Wasielewski +1
Normalization layers are ubiquitous in large language models (LLMs) yet represent a compute bottleneck: on hardware with distinct vector and matrix execution units, the RMS calcula…