19 citations · 44 across the 13 of their papers we have counts for
9 papers · 1 filter
Towards Learning High-Precision Least Squares Algorithms with Sequence Models
Jerry Liu, Jessica Grogan, Owen Dugan +4
This paper investigates whether sequence models can learn to perform numerical algorithms, e.g. gradient descent, on the fundamental problem of least squares. Our goal is to inheri…
Systems and Algorithms for Convolutional Multi-Hybrid Language Models at Scale
Jerome Ku, Eric Nguyen, David W. Romero +13
We introduce convolutional multi-hybrid architectures, with a design grounded on two simple observations. First, operators in hybrid models can be tailored to token manipulation ta…
KernelBench: Can LLMs Write Efficient GPU Kernels?
Anne Ouyang, Simon Guo, Simran Arora +4
Efficient GPU kernels are crucial for building performant machine learning architectures, but writing them is a time-consuming challenge that requires significant expertise; theref…
CodeMonkeys: Scaling Test-Time Compute for Software Engineering
Ryan Ehrlich, Bradley Brown, Jordan Juravsky +3
Scaling test-time compute is a promising axis for improving LLM capabilities. However, test-time compute can be scaled in a variety of ways, and effectively combining different app…
Scaling Laws for Precision
Tanishq Kumar, Zachary Ankner, Benjamin F. Spector +6
Low precision training and inference affect both the quality and cost of language models, but current scaling laws do not account for this. In this work, we devise "precision-aware…
ThunderKittens: Simple, Fast, and Adorable AI Kernels
Benjamin F. Spector, Simran Arora, Aaryan Singhal +2
The challenge of mapping AI architectures to GPU hardware is creating a critical bottleneck in AI progress. Despite substantial efforts, hand-written custom kernels fail to meet th…
LoLCATs: On Low-Rank Linearizing of Large Language Models
Michael Zhang, Simran Arora, Rahul Chalamala +5
Recent works show we can linearize large language models (LLMs) -- swapping the quadratic attentions of popular Transformer-based LLMs with subquadratic analogs, such as linear att…
Restructuring Vector Quantization with the Rotation Trick
Christopher Fifty, Ronald G. Junkins, Dennis Duan +5
Vector Quantized Variational AutoEncoders (VQ-VAEs) are designed to compress a continuous input to a discrete latent space and reconstruct it with minimal distortion. They operate…
Large Language Monkeys: Scaling Inference Compute with Repeated Sampling
Bradley Brown, Jordan Juravsky, Ryan Ehrlich +4
Scaling the amount of compute used to train language models has dramatically improved their capabilities. However, when it comes to inference, we often limit models to making only…