Visualizing probabilistic models in Minkowski space with intensive symmetrized Kullback-Leibler embedding

Abstract

We show that the predicted probability distributions for any $N$-parameter statistical model taking the form of an exponential family can be explicitly and analytically embedded isometrically in a $N+N$-dimensional Minkowski space. That is, the model predictions can be visualized as control parameters are varied, preserving the natural distance between probability distributions. All pairwise distances between model instances are given by the symmetrized Kullback-Leibler divergence. We give formulas for these intensive symmetrized Kullback Leibler (isKL) coordinate embeddings, and illustrate the resulting visualizations with the Bernoulli (coin toss) problem, the ideal gas, $n$ sided die, the nonlinear least squares fit, and the Gaussian fit. We highlight how isKL can be used to determine the minimum number of parameters needed to describe probabilistic data, and conclude by visualizing the prediction space of the two-dimensional Ising model, where we examine the manifold behavior near its critical point.

arXiv comment18 pages, 9 figures

References in corpus (13)

• Universally Sloppy Parameter Sensitivities in Systems Biology
• Sustaining the Internet with Hyperbolic Mapping
• Parameter Space Compression Underlies Emergent Theories and Predictive Models
• The geometry of nonlinear least squares with applications to sloppy models and optimization
• Why are nonlinear fits so challenging?
• The sloppy model universality class and the Vandermonde matrix
• Stochastic time-evolution, information geometry and the Cramer-Rao Bound
• On the Chi square and higher-order Chi distances for approximating f-divergences
• A dissipation bound for thermodynamic control
• Cluster representations and the Wolff algorithm in arbitrary external fields
• Visualizing probabilistic models: Intensive Principal Component Analysis
• Chebyshev approximation and the global geometry of sloppy models
• Online storage ring optimization using dimension-reduction and genetic algorithms

Cited by in corpus (4)

• The Training Process of Many Deep Networks Explores the Same Low-Dimensional Manifold
• Applications of information geometry to spiking neural network behavior
• Geometric Foundation of Nonequilibrium Transport: A Minkowski Embedding of Markov Dynamics
• Irreversibility as divergence from equilibrium