The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initialization

Mufan Bill Li, Mihai Nica, Daniel M. Roy

The logit outputs of a feedforward neural network at initialization are conditionally Gaussian, given a random covariance matrix defined by the penultimate layer. In this work, we…