The Fine Line: Navigating Large Language Model Pretraining with Down-streaming Capability Analysis

Chen Yang, Junzhuo Li, Xinyao Niu +11

Uncovering early-stage metrics that reflect final model performance is one core principle for large-scale pretraining. The existing scaling law demonstrates the power-law correlati…