Horizon-Independent Optimal Prediction with Log-Loss in Exponential Families
arXiv:1305.4324
Abstract
We study online learning under logarithmic loss with regular parametric models. Hedayati and Bartlett (2012b) showed that a Bayesian prediction strategy with Jeffreys prior and sequential normalized maximum likelihood (SNML) coincide and are optimal if and only if the latter is exchangeable, and if and only if the optimal strategy can be calculated without knowing the time horizon in advance. They put forward the question what families have exchangeable SNML strategies. This paper fully answers this open problem for one-dimensional exponential families. The exchangeability can happen only for three classes of natural exponential family distributions, namely the Gaussian, Gamma, and the Tweedie exponential family of order 3/2. Keywords: SNML Exchangeability, Exponential Family, Online Learning, Logarithmic Loss, Bayesian Strategy, Jeffreys Prior, Fisher Information1
23 pages
Cited by in corpus (6)
- Testing Order Constraints: Qualitative Differences Between Bayes Factors and Normalized Maximum Likelihood
- Divergence and Sufficiency for Convex Optimization
- Maximum Entropy and Sufficiency
- Bayesian Properties of Normalized Maximum Likelihood and its Fast Computation
- Large Alphabet Compression and Predictive Distributions through Poissonization and Tilting
- On the generalization of bayesian deep nets for multi-class classification