The Model Shape Behind Scaling Laws: A Gradient Superposition Perspective

Wenjie Sun, Jinning Yang, Shuai Zhang +1

Neural scaling laws guide resource allocation, but how should a fixed parameter budget be divided between depth and width? Motivated by inverse-width representation-error scaling u…