Publications (6)
TensorBench: Benchmarking Coding Agents on a Compiler-Based Tensor Framework
Bobby Yan, Fredrik Kjolstad
Repository-level coding benchmarks face a trade-off between task difficulty and evaluation reliability: tasks that challenge frontier models often involve large codebases with inco…
Scorch: A Library for Sparse Deep Learning
Bobby Yan, Alexander J. Root, Trevor Gale +2
The rapid growth in the size of deep learning models strains the capabilities of traditional dense computation paradigms. Leveraging sparse computation has become increasingly popu…
Partitioning Unstructured Sparse Tensor Algebra for Load-Balanced Parallel Execution
Atharva Chougule, Alexander J Root, Rubens Lacouture +3
Sparse tensor algebra is challenging to efficiently parallelize due to the irregular, data-dependent, and potentially skewed structure of sparse computation. We propose the first p…
FORML: Learning to Reweight Data for Fairness
Bobby Yan, Skyler Seto, Nicholas Apostoloff
Machine learning models are trained to minimize the mean loss for a single metric, and thus typically do not consider fairness and robustness. Neglecting such metrics in training c…
Hindsight Logging for Model Training
Rolando Garcia, Eric Liu, Vikram Sreekanti +5
In modern Machine Learning, model training is an iterative, experimental process that can consume enormous computation resources and developer time. To aid in that process, experie…
Holistic Evaluation of Language Models
Percy Liang, Rishi Bommasani, Tony Lee +47
Language models (LMs) are becoming the foundation for almost all major language technologies, but their capabilities, limitations, and risks are not well understood. We present Hol…