Optimizing F-measure: A Tale of Two Approaches

Abstract

F-measures are popular performance metrics, particularly for tasks with imbalanced data sets. Algorithms for learning to maximize F-measures follow two approaches: the empirical utility maximization (EUM) approach learns a classifier having optimal performance on training data, while the decision-theoretic approach learns a probabilistic model and then predicts labels with maximum expected F-measure. In this paper, we investigate the theoretical justifications and connections for these two approaches, and we study the conditions under which one approach is preferable to the other using synthetic and real datasets. Given accurate models, our results suggest that the two approaches are asymptotically equivalent given large training and test sets. Nevertheless, empirically, the EUM approach appears to be more robust against model misspecification, and given a good model, the decision-theoretic approach appears to be better for handling rare classes and a common domain adaptation scenario.

ICML2012

References in corpus (1)

• Introduction to the CoNLL-2002 Shared Task: Language-Independent Named Entity Recognition

Cited by in corpus (18)

• Learning Surrogate Losses
• A study on cost behaviors of binary classification measures in class-imbalanced problems
• Optimizing Non-decomposable Performance Measures: A Tale of Two Classes
• Online and Stochastic Gradient Methods for Non-decomposable Loss Functions
• Optimizing the F-measure for Threshold-free Salient Object Detection
• Learning Context-Dependent Choice Functions
• Optimizing Black-box Metrics with Adaptive Surrogates
• Top-K Pairwise Ranking: Bridging the Gap Among Ranking-Based Measures for Multi-Label Classification
• Optimal Decision-Theoretic Classification Using Non-Decomposable Performance Metrics
• Rethinking and Reweighting the Univariate Losses for Multi-Label Ranking: Consistency and Generalization
• Optimizing Generalized Rate Metrics through Game Equilibrium
• Optimizing Black-box Metrics with Iterative Example Weighting
• F-measure Maximization in Multi-Label Classification with Conditionally Independent Label Subsets
• F-measure Maximizing Logistic Regression
• A Minimax Probability Machine for Non-Decomposable Performance Measures
• Online Classification with Complex Metrics
• Training Over-parameterized Models with Non-decomposable Objectives
• Optimal rates for F-score binary classification