Divergence and Sufficiency for Convex Optimization
arXiv:1701.01010 · doi:10.3390/e19050206
Abstract
Logarithmic score and information divergence appear in information theory, statistics, statistical mechanics, and portfolio theory. We demonstrate that all these topics involve some kind of optimization that leads directly to regret functions and such regret functions are often given by a Bregman divergence. If the regret function also fulfills a sufficiency condition it must be proportional to information divergence. We will demonstrate that sufficiency is equivalent to the apparently weaker notion of locality and it is also equivalent to the apparently stronger notion of monotonicity. These sufficiency conditions have quite different relevance in the different areas of application, and often they are not fulfilled. Therefore sufficiency conditions can be used to explain when results from one area can be transferred directly to another and when one will experience differences.
arXiv comment39 pages, 3 figures
References in corpus (20)
- The information bottleneck method
- Game theory, maximum entropy, minimum discrepancy and robust Bayesian decision theory
- Monotonicity of quantum relative entropy revisited
- Higher-order interference and single-system postulates characterizing quantum theory
- Constructor Theory of Information
- Monotonicity of the Quantum Relative Entropy Under Positive Maps
- Sufficiency in quantum statistical inference
- Relative Entropy Bounds on Quantum, Private and Repeater Capacities
- Thermodynamics and the structure of quantum theory
- Ruling out higher-order interference from purity principles
- Proper local scoring rules on discrete sample spaces
- Preservation of a quantum Renyi relative entropy implies existence of a recovery map
- Information Measures: the Curious Case of the Binary Alphabet
- On the joint convexity of the Bregman divergence of matrices
- Horizon-Independent Optimal Prediction with Log-Loss in Exponential Families
- Characterisation of matrix entropies
- Sufficiency on the Stock Market
- Proper Scoring Rules and Bregman Divergences
- Quantum Information on Spectral Sets
- Proper Scoring and Sufficiency