Game theory, maximum entropy, minimum discrepancy and robust Bayesian decision theory

Abstract

We describe and develop a close relationship between two problems that have customarily been regarded as distinct: that of maximizing entropy, and that of minimizing worst-case expected loss. Using a formulation grounded in the equilibrium theory of zero-sum games between Decision Maker and Nature, these two problems are shown to be dual to each other, the solution to each providing that to the other. Although Topsœdescribed this connection for the Shannon entropy over 20 years ago, it does not appear to be widely known even in that important special case. We here generalize this theory to apply to arbitrary decision problems and loss functions. We indicate how an appropriate generalized definition of entropy can be associated with such a problem, and we show that, subject to certain regularity conditions, the above-mentioned duality continues to apply in this extended context. This simultaneously provides a possible rationale for maximizing entropy and a tool for finding robust Bayes acts. We also describe the essential identity between the problem of maximizing entropy and that of minimizing a related discrepancy or divergence between distributions. This leads to an extension, to arbitrary discrepancies, of a well-known minimax theorem for the case of Kullback-Leibler divergence (the ``redundancy-capacity theorem'' of information theory). For the important case of families of distributions having certain mean values specified, we develop simple sufficient conditions and methods for identifying the desired solutions.

Published by the Institute of Mathematical Statistics (http://www.imstat.org) in the Annals of Statistics (http://www.imstat.org/aos/) at http://dx.doi.org/10.1214/009053604000000553

Cited by in corpus (109)

• A tutorial introduction to the minimum description length principle
• An introduction to domain adaptation and transfer learning
• Distributionally Robust Optimization: A Review
• Composite Binary Losses
• Maximum likelihood estimation in log-linear models
• Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness
• Principles of Bayesian Inference using General Divergence Criteria
• The q-exponential family in statistical physics
• Generalised exponential families and associated entropy functions
• A Minimax Approach to Supervised Learning
• Reinforcement Learning Control of a Biomechanical Model of the Upper Extremity
• $\kappa$-exponential models from the geometrical viewpoint
• Fairness for Robust Log Loss Classification
• Boosting in the presence of outliers: adaptive classification with non-convex loss functions
• Bayesian Model Selection Based on Proper Scoring Rules
• If MaxEnt RL is the Answer, What is the Question?
• A Theory of Usable Information Under Computational Constraints
• Maximum Entropy RL (Provably) Solves Some Robust RL Problems
• Inherent Tradeoffs in Learning Fair Representations
• Timely Communications for Remote Inference
• Learning and Communications Co-Design for Remote Inference Systems: Feature Length Selection and Transmission Scheduling
• Coherent frequentism
• Linear Estimating Equations for Exponential Families with Application to Gaussian Linear Concentration Models
• Bregman divergence as general framework to estimate unnormalized statistical models
• Generalized minimizers of convex integral functionals, Bregman distance, Pythagorean identities
• A Unifying Causal Framework for Analyzing Dataset Shift-stable Learning Algorithms
• Learning Optimal Representations with the Decodable Information Bottleneck
• Geometric study for the Legendre duality of generalized entropies and its application to the porous medium equation
• Evaluating probability forecasts
• Resolving conflicts between statistical methods by probability combination: Application to empirical Bayes analyses of genomic data
• Maximum Entropy on Compact Groups
• Low Impact Artificial Intelligences
• Modeling the correlated activity of neural populations: A review
• Bayesian Predictive Densities Based on Latent Information Priors
• Controlling the degree of caution in statistical inference with the Bayesian and frequentist approaches as opposite extremes
• Preventing Failures Due to Dataset Shift: Learning Predictive Models That Transport
• From Proper Scoring Rules to Max-Min Optimal Forecast Aggregation
• The q-exponential family in statistical physics
• Properization: Constructing Proper Scoring Rules via Bayes Acts
• Asymptotically minimax Bayesian predictive densities for multinomial models
• Notions of the ergodic hierarchy for curved statistical manifolds
• Comparing Sequential Forecasters
• Maximum Entropy and Sufficiency
• Divergence and Sufficiency for Convex Optimization
• Data Set Models and Exponential Families in Statistical Physics and Beyond
• Non-extensivity of the configurational density distribution in the classical microcanonical ensemble
• Learning in the Presence of Corruption
• Information Geometry of q-Gaussian Densities and Behaviors of Solutions to Related Diffusion Equations
• Robust Covariate Shift Prediction with General Losses and Feature Views
• Learning with Fenchel-Young Losses
• Robust Regression for Safe Exploration in Control
• Follow-the-Regularized-Leader Routes to Chaos in Routing Games
• Second-order optimisation strategies for neural network quantum states
• Affine Invariant Divergences associated with Composite Scores and its Applications
• The Minimum Information Principle for Discriminative Learning
• Generalised Lipschitz Regularisation Equals Distributional Robustness
• Minimax Classification with 0-1 Loss and Performance Guarantees
• Fast rates in statistical and online learning
• Minimum Description Length Principle for Maximum Entropy Model Selection
• Compatible Weighted Proper Scoring Rules
• Robust Probability Updating
• Linearly Solvable Mean-Field Traffic Routing Games
• Information, Divergence and Risk for Binary Experiments
• The statistical strength of nonlocality proofs
• Blending Bayesian and frequentist methods according to the precision of prior information with an application to hypothesis testing
• A Survey of Learning Criteria Going Beyond the Usual Risk
• Continuity of Generalized Entropy and Statistical Learning
• Non-exponentially weighted aggregation: regret bounds for unbounded loss functions
• Active Learning in Video Tracking
• Elements of asymptotic theory with outer probability measures
• General Supervision via Probabilistic Transformations
• Consistent Robust Adversarial Prediction for General Multiclass Classification
• Learning Classifiers with Fenchel-Young Losses: Generalized Entropies, Margins, and Algorithms
• Target contrastive pessimistic risk for robust domain adaptation
• Information Aggregation in Exponential Family Markets
• Predictive Complexity and Generalized Entropy Rate of Stationary Ergodic Processes
• The exponential family in abstract information theory
• Generalized Maximum Causal Entropy for Inverse Reinforcement Learning
• A Game-Theoretic Analysis of Updating Sets of Probabilities
• General Probabilistic Surface Optimization and Log Density Estimation
• A Theory of Feature Learning
• Kernel Robust Bias-Aware Prediction under Covariate Shift
• Composite Bayesian inference
• Existence and Uniqueness of Proper Scoring Rules
• Relating maximum entropy, resilient behavior and game-theoretic equilibrium feedback operators in multi-channel systems
• Discontinuities in the Maximum-Entropy Inference
• Sparse Continuous Distributions and Fenchel-Young Losses
• Building Trees for Probabilistic Prediction via Scoring Rules
• Two types of $q$-Gaussian distributions used to study the diffusion in a finite region
• The Age of Correlated Features in Supervised Learning based Forecasting
• Supervised classification via minimax probabilistic transformations
• Generalised Entropy MDPs and Minimax Regret
• Market Making with Decreasing Utility for Information
• Measuring Model Risk
• Supermartingales in Prediction with Expert Advice
• Generalized Entropy Concentration for Counts
• Le Cam meets LeCun: Deficiency and Generic Feature Learning
• Universality of Bayesian mixture predictors
• Preference elicitation and inverse reinforcement learning
• On Combinatorial Rectangles with Minimum $\infty$-Discrepancy
• Contrastive Pessimistic Likelihood Estimation for Semi-Supervised Classification
• Update of prior probabilities by minimal divergence
• Making Decisions Using Sets of Probabilities: Updating, Time Consistency, and Calibration
• Cooperative Hypothesis Testing by Two Observers with Asymmetric Information
• Entropy Concentration and the Empirical Coding Game
• A Game Theoretic Approach to Quantum Information
• Optimal Capital Injections with the Risk of Ruin: A Stochastic Differential Game of Impulse Control and Stopping Approach
• Distributionally Robust Graphical Models
• On Robust Pseudo-Bayes Estimation for the Independent Non-homogeneous Set-up