Revisiting the Scaling Properties of Downstream Metrics in Large Language Model Training

Jakub Krajewski, Amitis Shidani, Dan Busbridge +2

While scaling laws for Large Language Models (LLMs) traditionally focus on proxy metrics like pretraining loss, predicting downstream task performance has been considered unreliabl…