14 citations · 16 across the 6 of their papers we have counts for
6 papers
ParallelKittens: Systematic and Practical Simplification of Multi-GPU AI Kernels
Stuart H. Sul, Simran Arora, Benjamin F. Spector +1
Inter-GPU communication has become a major bottleneck for modern AI workloads as models scale and improvements in hardware compute throughput outpace improvements in interconnect b…
Scaling Laws for Precision
Tanishq Kumar, Zachary Ankner, Benjamin F. Spector +6
Low precision training and inference affect both the quality and cost of language models, but current scaling laws do not account for this. In this work, we devise "precision-aware…
ThunderKittens: Simple, Fast, and Adorable AI Kernels
Benjamin F. Spector, Simran Arora, Aaryan Singhal +2
The challenge of mapping AI architectures to GPU hardware is creating a critical bottleneck in AI progress. Despite substantial efforts, hand-written custom kernels fail to meet th…
LoLCATs: On Low-Rank Linearizing of Large Language Models
Michael Zhang, Simran Arora, Rahul Chalamala +5
Recent works show we can linearize large language models (LLMs) -- swapping the quadratic attentions of popular Transformer-based LLMs with subquadratic analogs, such as linear att…
Just read twice: closing the recall gap for recurrent language models
Simran Arora, Aman Timalsina, Aaryan Singhal +6
Recurrent large language models that compete with Transformers in language modeling perplexity are emerging at a rapid rate (e.g., Mamba, RWKV). Excitingly, these architectures use…
Monarch Mixer: A Simple Sub-Quadratic GEMM-Based Architecture
Daniel Y. Fu, Simran Arora, Jessica Grogan +7
Machine learning models are increasingly being scaled in both sequence length and model dimension to reach longer contexts and better performance. However, existing architectures s…