Posterior Dispersion Indices
arXiv:1605.07604
Abstract
Probabilistic modeling is cyclical: we specify a model, infer its posterior, and evaluate its performance. Evaluation drives the cycle, as we revise our model based on how it performs. This requires a metric. Traditionally, predictive accuracy prevails. Yet, predictive accuracy does not tell the whole story. We propose to evaluate a model through posterior dispersion. The idea is to analyze how each datapoint fares in relation to posterior uncertainty around the hidden structure. We propose a family of posterior dispersion indices (PDI) that capture this idea. A PDI identifies rich patterns of model mismatch in three real data examples: voting preferences, supermarket shopping, and population genetics.
References in corpus (6)
- Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC
- Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory
- Comparison of Bayesian predictive methods for model selection
- Automatic Variational Inference in Stan
- Bayesian Cross Validation and WAIC for Predictive Prior Design in Regular Asymptotic Theory
- A Unified Treatment of Predictive Model Comparison