Relative-Based Scaling Law for Neural Language Models

Baoqing Yue, Jinyuan Zhou, Zixi Wei +3

Scaling laws aim to accurately predict model performance across different scales. Existing scaling-law studies almost exclusively rely on cross-entropy as the evaluation metric. Ho…