55 citations · 102 across the 14 of their papers we have counts for
13 papers · 1 filter
Deterministic Bounds and Random Estimates of Metric Tensors on Neuromanifolds
Ke Sun
The high-dimensional parameter space of deep neural networks -- the neuromanifold -- is endowed with a unique metric tensor defined by the Fisher information. Reliable and scalable…
Trade-Offs of Diagonal Fisher Information Matrix Estimators
Alexander Soen, Ke Sun
The Fisher information matrix can be used to characterize the local geometry of the parameter space of neural networks. It elucidates insightful theories and useful tools to unders…
Transformed Distribution Matching for Missing Value Imputation
He Zhao, Ke Sun, Amir Dezfouli +1
We study the problem of imputing missing values in a dataset, which has important applications in many domains. The key to missing value imputation is to capture the data distribut…
On the Variance of the Fisher Information for Deep Learning
Alexander Soen, Ke Sun
In the realm of deep learning, the Fisher information matrix (FIM) gives novel insights and useful tools to characterize the loss landscape, perform second-order optimization, and…
A Geometric Modeling of Occam's Razor in Deep Learning
Ke Sun, Frank Nielsen
Why do deep neural networks (DNNs) benefit from very high dimensional parameter spaces? Their huge parameter complexities vs stunning performance in practice is all the more intrig…
Fisher-Bures Adversary Graph Convolutional Networks
Ke Sun, Piotr Koniusz, Zhen Wang
In a graph convolutional network, we assume that the graph is generated wrt some observation noise. During learning, we make small random perturbations of the graph and tr…
On The Chain Rule Optimal Transport Distance
Frank Nielsen, Ke Sun
We define a novel class of distances between statistical multivariate distributions by modeling an optimal transport problem on their marginals with respect to a ground distance de…
Intrinsic Universal Measurements of Non-linear Embeddings
Ke Sun
A basic problem in machine learning is to find a mapping from a low dimensional latent space to a high dimensional observation space . Modern tools s…
Guaranteed Deterministic Bounds on the Total Variation Distance between Univariate Mixtures
Frank Nielsen, Ke Sun
The total variation distance is a core statistical distance between probability measures that satisfies the metric axioms, with value always falling in . This distance plays…
Clustering in Hilbert simplex geometry
Frank Nielsen, Ke Sun
Clustering categorical distributions in the finite-dimensional probability simplex is a fundamental task met in many applications dealing with normalized histograms. Traditionally,…
Coarse Grained Exponential Variational Autoencoders
Ke Sun, Xiangliang Zhang
Variational autoencoders (VAE) often use Gaussian or category distribution to model the inference process. This puts a limit on variational learning because this simplified assumpt…
Relative Natural Gradient for Learning Large Complex Models
Ke Sun, Frank Nielsen
Fisher information and natural gradient provided deep insights and powerful tools to artificial neural networks. However related analysis becomes more and more difficult as the lea…
Guaranteed bounds on the Kullback-Leibler divergence of univariate mixtures using piecewise log-sum-exp inequalities
Frank Nielsen, Ke Sun
Information-theoretic measures such as the entropy, cross-entropy and the Kullback-Leibler divergence between two mixture models is a core primitive in many signal processing tasks…