Amortizing Scaling Law Construction Costs

Abhash Kumar Jha, Diana Alexandra Onuţu, Neeratyoy Mallik +6

Scaling laws guide the design choices for training large foundation models, but deriving them involves training an exhaustive grid over hyperparameters, token budgets, and paramete…