Can Pre-training Indicators Reliably Predict Fine-tuning Outcomes of LLMs?

Hansi Zeng, Kai Hui, Honglei Zhuang +4

While metrics available during pre-training, such as perplexity, correlate well with model performance at scaling-laws studies, their predictive capacities at a fixed model size re…