Neyman-Pearson (NP) classification algorithms and NP receiver operating characteristics (NP-ROC)

Abstract

In many binary classification applications such as disease diagnosis and spam detection, practitioners often face great needs to control type I errors (i.e., the conditional probability of misclassifying a class 0 observation as class 1) so that it remains below a desired threshold. To address this need, the Neyman-Pearson (NP) classification paradigm is a natural choice; it minimizes type II error (i.e., the conditional probability of misclassifying a class 1 observation as class 0) while enforcing an upper bound, $\alpha$, on the type I error. Although the NP paradigm has a century-long history in hypothesis testing, it has not been well recognized and implemented in classification schemes. Common practices that directly limit the empirical type I error to no more than $\alpha$ do not satisfy the type I error control objective because the resulting classifiers are still likely to have type I errors much larger than $\alpha$. As a result, the NP paradigm has not been properly implemented for many classification scenarios in practice. In this work, we develop the first umbrella algorithm that implements the NP paradigm for all scoring-type classification methods, including popular methods such as logistic regression, support vector machines and random forests. Powered by this umbrella algorithm, we propose a novel graphical tool for NP classification methods: NP receiver operating characteristic (NP-ROC) bands, motivated by the popular receiver operating characteristic (ROC) curves. NP-ROC bands will help choose $\alpha$ in a data adaptive way and compare different NP classifiers. We demonstrate the use and properties of the NP umbrella algorithm and NP-ROC bands, available in the R package nproc, through simulation and real data case studies.

References in corpus (5)

• Assessing Technical Performance in Differential Gene Expression Experiments with External Spike-in RNA Control Ratio Mixtures
• Neyman-Pearson classification, convexity and stochastic constraints
• Multiclass Sparse Discriminant Analysis
• TROM: A Testing-based Method for Finding Transcriptomic Similarity of Biological Samples
• Neyman-Pearson Classification under High-Dimensional Settings

Cited by in corpus (17)

• TROM: A Testing-based Method for Finding Transcriptomic Similarity of Biological Samples
• Intentional Control of Type I Error over Unconscious Data Distortion: a Neyman-Pearson Approach to Text Classification
• Instance-Based Classification through Hypothesis Testing
• Neyman-Pearson Multi-class Classification via Cost-sensitive Learning
• A Generalized Neyman-Pearson Criterion for Optimal Domain Adaptation
• THORS: An Efficient Approach for Making Classifiers Cost-sensitive
• Meta-Cal: Well-controlled Post-hoc Calibration by Ranking
• A Neural Network Approach for Online Nonlinear Neyman-Pearson Classification
• Bridging Cost-sensitive and Neyman-Pearson Paradigms for Asymmetric Binary Classification
• Statistical hypothesis testing versus machine-learning binary classification: distinctions and guidelines
• Introduction and analysis of a method for the investigation of QCD-like tree data
• Non-splitting Neyman-Pearson Classifiers
• Imbalanced classification: a paradigm-based review
• A flexible model-free prediction-based framework for feature ranking
• Exploring the Possibility of a Recovery of Physics Process Properties from a Neural Network Model
• How to Control the Error Rates of Binary Classifiers
• Quantum phase classification via partial tomography-based quantum hypothesis testing