Approximate Spectral Decomposition of Fisher Information Matrix for Simple ReLU Networks

Yoshinari Takeishi, Masazumi Iida, Jun'ichi Takeuchi

We argue the Fisher information matrix (FIM) of one hidden layer networks with the ReLU activation function. For a network, let $W$ denote the $d\times p$ weight matrix from the $…