Maximum Entropy and Bayesian Data Analysis: Entropic Priors

Abstract

The problem of assigning probability distributions which objectively reflect the prior information available about experiments is one of the major stumbling blocks in the use of Bayesian methods of data analysis. In this paper the method of Maximum (relative) Entropy (ME) is used to translate the information contained in the known form of the likelihood into a prior distribution for Bayesian inference. The argument is inspired and guided by intuition gained from the successful use of ME methods in statistical mechanics. For experiments that cannot be repeated the resulting "entropic prior" is formally identical with the Einstein fluctuation formula. For repeatable experiments, however, the expected value of the entropy of the likelihood turns out to be relevant information that must be included in the analysis. The important case of a Gaussian likelihood is treated in detail.

23 pages, 2 figures

Cited by in corpus (43)

• Maximum Caliber: a general variational principle for dynamical systems
• Beyond Boltzmann-Gibbs statistics: Maximum entropy hyperensembles out-of-equilibrium
• Jacobi Fields on Statistical Manifolds of Negative Curvature
• Works on an information geometrodynamical approach to chaos
• Lectures on Probability, Entropy, and Statistical Physics
• Essential equivalence of the GENERIC and Steepest Entropy Ascent models of dissipation for non-equilibrium thermodynamics
• Entropy, Information, and the Updating of Probabilities
• Poroelasticity as a Model of Soft Tissue Structure: Hydraulic Permeability Inference for Magnetic Resonance Elastography in Silico
• The Effect Of Microscopic Correlations On The Information Geometric Complexity Of Gaussian Statistical Models
• Quantifying The Complexity Of Geodesic Paths On Curved Statistical Manifolds Through Information Geometric Entropies and Jacobi Fields
• Can chaotic quantum energy levels statistics be characterized using information geometry and inference methods?
• A maximum entropy thermodynamics of small systems
• Quantifying extrinsic noise in gene expression using the maximum entropy framework
• Information-Geometric Indicators of Chaos in Gaussian Models on Statistical Manifolds of Negative Ricci Curvature
• Information Geometry, Inference Methods and Chaotic Energy Levels Statistics
• Cell responses only partially shape cell-to-cell variations in protein abundances in Escherichia coli chemotaxis
• Pairwise maximum-entropy models and their Glauber dynamics: bimodality, bistability, non-ergodicity problems, and their elimination via inhibition
• Intermediate-Mass Black Holes in binary-rich star clusters
• The luminosity function of quasars by the Principle of Maximum Entropy
• Information Geometry of Quantum Entangled Gaussian Wave-Packets
• Using Relative Entropy to Find Optimal Approximations: an Application to Simple Fluids
• Information geometric complexity of a trivariate Gaussian statistical model
• Geometrodynamics of Information on Curved Statistical Manifolds and its Applications to Chaos
• Conditional maximum-entropy method for selecting prior distributions in Bayesian statistics
• On The Complexity Of Statistical Models Admitting Correlations
• Aspects of geodesical motion with Fisher-Rao metric: classical and quantum
• On the instability of two entropic dynamical models
• On the role of composition entropies in the statistical mechanics of polydisperse systems
• Complexity Characterization in a Probabilistic Approach to Dynamical Systems Through Information Geometry and Inductive Inference
• The Census and the Second Law: An Entropic Approach to Optimal Apportionment for the U.S. House of Representatives
• Nonextensive Pythagoras' Theorem
• A Thermodynamical Approach for Probability Estimation
• Inferring transition rates on networks with incomplete knowledge
• Local quantum information dynamics
• Avoiding subtraction and division of stochastic signals using normalizing flows: NFdeconvolve
• Opinions and dark energy: A mechanism of long-range repulsion in a toy opinion particle model
• Softening the Complexity of Entropic Motion on Curved Statistical Manifolds
• Detecting temperature fluctuations at equilibrium
• Reexamination of an Information Geometric Construction of Entropic Indicators of Complexity
• The Information Geometry of Chaos
• Overcoming computational inability to predict clinical outcome from high-dimensional patient data using Bayesian methods
• The Brandeis Dice Problem and Statistical Mechanics
• The Entropic Dynamics of Relativistic Quantum Fields in Curved Space-time