2 citations · 3 across the 3 of their papers we have counts for
5 papers · 1 filter
The Spectrum of Fisher Information of Deep Networks Achieving Dynamical Isometry
Tomohiro Hayase, Ryo Karakida
The Fisher information matrix (FIM) is fundamental to understanding the trainability of deep neural nets (DNN), since it describes the parameter space's local metric. We investigat…
Pathological spectra of the Fisher information metric and its variants in deep neural networks
Ryo Karakida, Shotaro Akaho, Shun-ichi Amari
The Fisher information matrix (FIM) plays an essential role in statistics and machine learning as a Riemannian metric tensor or a component of the Hessian matrix of loss functions.…
The Normalization Method for Alleviating Pathological Sharpness in Wide Neural Networks
Ryo Karakida, Shotaro Akaho, Shun-ichi Amari
Normalization methods play an important role in enhancing the performance of deep learning while their theoretical understandings have been limited. To theoretically elucidate the…
Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach
Ryo Karakida, Shotaro Akaho, Shun-ichi Amari
The Fisher information matrix (FIM) is a fundamental quantity to represent the characteristics of a stochastic model, including deep neural networks (DNNs). The present study revea…
Concept Formation and Dynamics of Repeated Inference in Deep Generative Models
Yoshihiro Nagano, Ryo Karakida, Masato Okada
Deep generative models are reported to be useful in broad applications including image generation. Repeated inference between data space and latent space in these models can denois…