A Hitchhiker's Guide to Scaling Law Estimation

Leshem Choshen, Yang Zhang, Jacob Andreas

Scaling laws predict the loss of a target machine learning model by extrapolating from easier-to-train models with fewer parameters or smaller training sets. This provides an effic…