A Neural Scaling Law from the Dimension of the Data Manifold

Utkarsh Sharma, Jared Kaplan

When data is plentiful, the loss achieved by well-trained neural networks scales as a power-law $L\propto N^{-\alpha }$ in the number of network parameters $N$. This empirical scaling l…