Bayesian Nonparametric Models for Multiple Raters: a General Statistical Framework

Abstract

Rating procedure is crucial in many applied fields (e.g., educational, clinical, emergency). It implies that a rater (e.g., teacher, doctor) rates a subject (e.g., student, doctor) on a rating scale. Given raters variability, several statistical methods have been proposed for assessing and improving the quality of ratings. Model estimation in the presence of heterogeneity has been one of the recent challenges in this research line. Consequently, several methods have been proposed to address this issue under a parametric multilevel modelling framework, in which strong distributional assumptions are made. We propose a more flexible model under the Bayesian nonparametric (BNP) framework, in which most of those assumptions are relaxed. By eliciting hierarchical discrete nonparametric priors, the model accommodates clusters among raters and subjects, naturally accounts for heterogeneity, and improves estimates accuracy. We propose a general BNP heteroscedastic framework to analyse continuous and coarse rating data and possible latent differences among subjects and raters. The estimated densities are used to make inferences about the rating process and the quality of the ratings. By exploiting a stick-breaking representation of the Dirichlet Process, a general class of Intraclass Correlation Coefficient (ICC) indices might be derived for these models. Our method allows us to independently identify latent similarities between subjects and raters and can be applied in precise education to improve personalised teaching programs or interventions. Theoretical results about the ICC are provided together with computational strategies. Simulations and a real-world application are presented, and possible future directions are discussed.

References in corpus (9)

• Nested Hierarchical Dirichlet Processes
• Bayesian cluster analysis: Point estimation and credible balls
• Discussion on Bayesian Cluster Analysis: Point Estimation and Credible Balls by Sara Wade and Zoubin Ghahramani
• Are Gibbs-type priors the most natural generalization of the Dirichlet process?
• Flexible clustering via hidden hierarchical Dirichlet priors
• Stochastic approximations to the Pitman-Yor process
• Assessing inter-rater reliability with heterogeneous variance components models: Flexible approach accounting for contextual variables
• Scalable Bayesian computation for crossed and nested hierarchical models
• Assessing quality of selection procedures: Lower bound of false positive rate as a function of inter-rater reliability