5 papers · 1 filter
HipKittens: Fast and Furious AMD Kernels
William Hu, Drew Wadsworth, Sean Siddens +6
AMD GPUs offer state-of-the-art compute and memory bandwidth; however, peak performance AMD kernels are written in raw assembly. To address the difficulty of mapping AI algorithms…
Towards Learning High-Precision Least Squares Algorithms with Sequence Models
Jerry Liu, Jessica Grogan, Owen Dugan +4
This paper investigates whether sequence models can learn to perform numerical algorithms, e.g. gradient descent, on the fundamental problem of least squares. Our goal is to inheri…
LoLCATs: On Low-Rank Linearizing of Large Language Models
Michael Zhang, Simran Arora, Rahul Chalamala +5
Recent works show we can linearize large language models (LLMs) -- swapping the quadratic attentions of popular Transformer-based LLMs with subquadratic analogs, such as linear att…
KernelBench: Can LLMs Write Efficient GPU Kernels?
Anne Ouyang, Simon Guo, Simran Arora +4
Efficient GPU kernels are crucial for building performant machine learning architectures, but writing them is a time-consuming challenge that requires significant expertise; theref…
ThunderKittens: Simple, Fast, and Adorable AI Kernels
Benjamin F. Spector, Simran Arora, Aaryan Singhal +2
The challenge of mapping AI architectures to GPU hardware is creating a critical bottleneck in AI progress. Despite substantial efforts, hand-written custom kernels fail to meet th…