Is the Number of Trainable Parameters All That Actually Matters?

Amélie Chatelain, Amine Djeghri, Daniel Hesslow +2

Recent work has identified simple empirical scaling laws for language models, linking compute budget, dataset size, model size, and autoregressive modeling loss. The validity of th…