4 citations · 8 across the 7 of their papers we have counts for
4 papers · 1 filter
Scaling Laws for Precision
Tanishq Kumar, Zachary Ankner, Benjamin F. Spector +6
Low precision training and inference affect both the quality and cost of language models, but current scaling laws do not account for this. In this work, we devise "precision-aware…
Critique-out-Loud Reward Models
Zachary Ankner, Mansheej Paul, Brandon Cui +2
Traditionally, reward models used for reinforcement learning from human feedback (RLHF) are trained to directly predict preference scores without leveraging the generation capabili…
Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models
Zachary Ankner, Cody Blakeney, Kartik Sreenivasan +3
In this work, we investigate whether small language models can determine high-quality subsets of large-scale text datasets that improve the performance of larger language models. W…
The Effect of Data Dimensionality on Neural Network Prunability
Zachary Ankner, Alex Renda, Gintare Karolina Dziugaite +2
Practitioners prune neural networks for efficiency gains and generalization improvements, but few scrutinize the factors determining the prunability of a neural network the maximum…