Approximating Simple ReLU Networks based on Spectral Decomposition of Fisher Information

Ka Long Keith Ho, Yoshinari Takeishi, Junichi Takeuchi

Properties of Fisher information matrices of 2-layer neural ReLU networks with random hidden weights are studied. For these networks, it is known that the eigenvalue distribution h…