4 papers
Learning Shrinks the Hard Tail: Training-Dependent Inference Scaling in a Solvable Linear Model
Noam Levi
We analyze neural scaling laws in a solvable model of last-layer fine-tuning where targets have intrinsic, instance-heterogeneous difficulty. In our Latent Instance Difficulty (LID…
Efficient Prediction of Pass@k Scaling in Large Language Models
Joshua Kazdan, Rylan Schaeffer, Youssef Allouah +4
Assessing the capabilities and risks of frontier AI systems is a critical area of research, and recent work has shown that repeated sampling from models can dramatically increase b…
Evaluating the Robustness of Chinchilla Compute-Optimal Scaling
Rylan Schaeffer, Noam Levi, Andreas Kirsch +4
Hoffman et al (2022)'s Chinchilla paper introduced the principle of compute-optimal scaling, laying a foundation for future scaling of language models. In the years since, however,…
Pretraining Scaling Laws for Generative Evaluations of Language Models
Rylan Schaeffer, Noam Levi, Brando Miranda +1
Neural scaling laws have driven the field's ever-expanding exponential growth in parameters, data and compute. While scaling behaviors for pretraining losses and discriminative ben…