Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networks

Greg Yang, Dingli Yu, Chen Zhu +1

By classifying infinite-width neural networks and identifying the *optimal* limit, Tensor Programs IV and V demonstrated a universal way, called $\mu$P, for *widthwise hyperparameter…