Utility-Theoretic Ranking for Semi-Automated Text Classification
arXiv:1503.00491 · doi:10.1145/2742548
Abstract
\emph{Semi-Automated Text Classification} (SATC) may be defined as the task of ranking a set of automatically labelled textual documents in such a way that, if a human annotator validates (i.e., inspects and corrects where appropriate) the documents in a top-ranked portion of with the goal of increasing the overall labelling accuracy of , the expected increase is maximized. An obvious SATC strategy is to rank so that the documents that the classifier has labelled with the lowest confidence are top-ranked. In this work we show that this strategy is suboptimal. We develop new utility-theoretic ranking methods based on the notion of \emph{validation gain}, defined as the improvement in classification effectiveness that would derive by validating a given automatically labelled document. We also propose a new effectiveness measure for SATC-oriented ranking methods, based on the expected reduction in classification error brought about by partially validating a list generated by a given ranking method. We report the results of experiments showing that, with respect to the baseline method above, and according to the proposed measure, our utility-theoretic ranking methods can achieve substantially higher expected reductions in classification error.
Forthcoming on ACM Transactions on Knowledge Discovery from Data