Relevance Judgment Convergence Degree -- A Measure of Inconsistency among Assessors for Information Retrieval

Dengya Zhu, Shastri L Nimmagadda, Kok Wai Wong +1

Relevance judgment of human assessors is inherently subjective and dynamic when evaluation datasets are created for Information Retrieval (IR) systems. However, a small group of ex…