Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation

Abstract

Commonly used evaluation measures including Recall, Precision, F-Measure and Rand Accuracy are biased and should not be used without clear understanding of the biases, and corresponding identification of chance or base case levels of the statistic. Using these measures a system that performs worse in the objective sense of Informedness, can appear to perform better under any of these commonly used measures. We discuss several concepts and measures that reflect the probability that prediction is informed versus chance. Informedness and introduce Markedness as a dual measure for the probability that prediction is marked versus chance. Finally we demonstrate elegant connections between the concepts of Informedness, Markedness, Correlation and Significance as well as their intuitive relationships with Recall and Precision, and outline the extension from the dichotomous case to the general multi-class case.

27 pages, 7 figures. Updated and fixed egregious formatting errors (including a table overlapping text) that were introduced by the publisher. This open access journal appears to have been discontinued. arXiv admin note: text overlap with arXiv:1504.00854

References in corpus (1)

• Assessing agreement on classification tasks: the kappa statistic

Cited by in corpus (220)

• U-Net and its variants for medical image segmentation: theory and applications
• Regularized Deep Networks in Intelligent Transportation Systems: A Taxonomy and a Case Study
• Fame for sale: efficient detection of fake Twitter followers
• explAIner: A Visual Analytics Framework for Interactive and Explainable Machine Learning
• Learning Open Set Network with Discriminative Reciprocal Points
• What is Wrong with Topic Modeling? (and How to Fix it Using Search-based Software Engineering)
• Social Fingerprinting: detection of spambot groups through DNA-inspired behavioral modeling
• Machine Learning for Intrusion Detection in Industrial Control Systems: Applications, Challenges, and Recommendations
• StackGenVis: Alignment of Data, Algorithms, and Models for Stacking Ensemble Learning Using Performance Metrics
• Hydra: an Ensemble of Convolutional Neural Networks for Geospatial Land Classification
• Real-Time Fruit Recognition and Grasping Estimation for Autonomous Apple Harvesting
• MAD-GAN: Multivariate Anomaly Detection for Time Series Data with Generative Adversarial Networks
• Assessing Software Defection Prediction Performance: Why Using the Matthews Correlation Coefficient Matters
• Deep Gated Recurrent and Convolutional Network Hybrid Model for Univariate Time Series Classification
• Metrics for Benchmarking and Uncertainty Quantification: Quality, Applicability, and a Path to Best Practices for Machine Learning in Chemistry
• Meaningless comparisons lead to false optimism in medical machine learning
• On the limits of cross-domain generalization in automated X-ray prediction
• Determining the Veracity of Rumours on Twitter
• Gastric histopathology image segmentation using a hierarchical conditional random field
• BIRADS Features-Oriented Semi-supervised Deep Learning for Breast Ultrasound Computer-Aided Diagnosis
• A Survey of Adversarial Learning on Graphs
• Understanding Environmental Posts: Sentiment and Emotion Analysis of Social Media Data
• Deep Semantic Segmentation in an AUV for Online Posidonia Oceanica Meadows identification
• Anomaly-Based Intrusion Detection by Machine Learning: A Case Study on Probing Attacks to an Institutional Network
• Estimating indoor occupancy through low-cost BLE devices
• A Machine-learning based Probabilistic Perspective on Dynamic Security Assessment
• The MCC-F1 curve: a performance evaluation technique for binary classification
• Comparative evaluation of instrument segmentation and tracking methods in minimally invasive surgery
• A Comprehensive Safety Metric to Evaluate Perception in Autonomous Systems
• Fair comparison of skin detection approaches on publicly available datasets
• Event-based Failure Prediction in Distributed Business Processes
• On the Feasibility of Generic Deep Disaggregation for Single-Load Extraction
• Detecting Privacy Requirements from User Stories with NLP Transfer Learning Models
• A Survey of Machine Learning Methods and Challenges for Windows Malware Classification
• TorchXRayVision: A library of chest X-ray datasets and models
• Camera-trap images segmentation using multi-layer robust principal component analysis
• Classifier uncertainty: evidence, potential impact, and probabilistic treatment
• Energy efficient distributed analytics at the edge of the network for IoT environments
• ESCORT: Ethereum Smart COntRacTs Vulnerability Detection using Deep Neural Network and Transfer Learning
• Concurrent Activity Recognition with Multimodal CNN-LSTM Structure
• Statistical Anomaly Detection via Composite Hypothesis Testing for Markov Models
• Combining Unsupervised and Supervised Learning for Asset Class Failure Prediction in Power Systems
• ArguLens: Anatomy of Community Opinions On Usability Issues Using Argumentation Models
• Approximate Message Passing with Restricted Boltzmann Machine Priors
• A semi-supervised methodology for fishing activity detection using the geometry behind the trajectory of multiple vessels
• Performance Analysis and Comparison of Machine and Deep Learning Algorithms for IoT Data Classification
• VisEvol: Visual Analytics to Support Hyperparameter Search through Evolutionary Optimization
• A XGBoost risk model via feature selection and Bayesian hyper-parameter optimization
• Maximum Relevance and Minimum Redundancy Feature Selection Methods for a Marketing Machine Learning Platform
• A new framework for optimal classifier design
• Identifying supportive student factors for mindset interventions: A two-model machine learning approach
• Lung cancer screening with low-dose CT scans using a deep learning approach
• Exploring Key Factors for Long-Term Vessel Incident Risk Prediction
• Features Fusion for Classification of Logos
• Multi-Scale Iterative Refinement Network for RGB-D Salient Object Detection
• Distributed Deep Forest and its Application to Automatic Detection of Cash-out Fraud
• Translocatome: a novel resource for the analysis of protein translocation between cellular organelles
• Generalizing AUC Optimization to Multiclass Classification for Audio Segmentation With Limited Training Data
• An Adaptive Feature Extraction Algorithm for Classification of Seismocardiographic Signals
• A model-driven approach for a new generation of adaptive libraries
• BEHRT: Transformer for Electronic Health Records
• Detecting Target-Area Link-Flooding DDoS Attacks using Traffic Analysis and Supervised Learning
• Guiding Cascading Failure Search with Interpretable Graph Convolutional Network
• SLAQ: Quality-Driven Scheduling for Distributed Machine Learning
• Deep Mangoes: from fruit detection to cultivar identification in colour images of mango trees
• A fast learning algorithm for One-Class Slab Support Vector Machines
• Non-invasive analysis of blood-brain barrier permeability based on wavelet and machine learning approaches
• Dandelion: Redesigning the Bitcoin Network for Anonymity
• A Bootstrap Machine Learning Approach to Identify Rare Disease Patients from Electronic Health Records
• Fairness Evaluation in Text Classification: Machine Learning Practitioner Perspectives of Individual and Group Fairness
• Layer-wise Characterization of Latent Information Leakage in Federated Learning
• The iMet Collection 2019 Challenge Dataset
• Credit card fraud detection using machine learning: A survey
• A Systematic Review of Unsupervised Learning Techniques for Software Defect Prediction
• Using generative adversarial networks to synthesize artificial financial datasets
• Repurpose Open Data to Discover Therapeutics for COVID-19 using Deep Learning
• Reservoir Computing for Detection of Steady State in Performance Tests of Compressors
• Towards a Near Universal Time Series Data Mining Tool: Introducing the Matrix Profile
• Prior choice affects ability of Bayesian neural networks to identify unknowns
• Automated Essay Scoring Using Transformer Models
• A Federated Semi-Supervised Learning Approach for Network Traffic Classification
• Correlations Between Human Mobility and Social Interaction Reveal General Activity Patterns
• Deep Sequence Models for Text Classification Tasks
• FERA 2017 - Addressing Head Pose in the Third Facial Expression Recognition and Analysis Challenge
• Relative Pairwise Relationship Constrained Non-negative Matrix Factorisation
• An ADMM Approach to Masked Signal Decomposition Using Subspace Representation
• Symbolic Representation and Classification of Logos
• A Survey on Human Machine Interaction in Industry 4.0
• Automatic Lesion Boundary Segmentation in Dermoscopic Images with Ensemble Deep Learning Methods
• Detecting Autism Spectrum Disorders with Machine Learning Models Using Speech Transcripts
• Recurrent Meta-Structure for Robust Similarity Measure in Heterogeneous Information Networks
• Brain Topography Adaptive Network for Satisfaction Modeling in Interactive Information Access System
• B-HAR: an open-source baseline framework for in depth study of human activity recognition datasets and workflows
• Statistical Evaluation of Anomaly Detectors for Sequences
• Center-Extraction-Based Three Dimensional Nuclei Instance Segmentation of Fluorescence Microscopy Images
• The impact of using biased performance metrics on software defect prediction research
• Learning Context-Dependent Choice Functions
• High Enough? Explaining and Predicting Traveler Satisfaction Using Airline Review
• Leveraging Contextual Cues for Generating Basketball Highlights
• Re-scale AdaBoost for Attack Detection in Collaborative Filtering Recommender Systems
• CRISLoc: Reconstructable CSI Fingerprintingfor Indoor Smartphone Localization
• EPP: interpretable score of model predictive power
• Artificial Intelligence Based Malware Analysis
• Automatic Arabic Dialect Identification Systems for Written Texts: A Survey
• Partial Discharge Detection on Aerial Covered Conductors Using Time-Series Decomposition and Long Short-term Memory Network
• It's Moving! A Probabilistic Model for Causal Motion Segmentation in Moving Camera Videos
• Method of noun phrase detection in Ukrainian texts
• A New Clustering-Based Technique for the Acceleration of Deep Convolutional Networks
• Multivariate Arrival Times with Recurrent Neural Networks for Personalized Demand Forecasting
• A Comparison for Anti-noise Robustness of Deep Learning Classification Methods on a Tiny Object Image Dataset: from Convolutional Neural Network to Visual Transformer and Performer
• The wisdom of the few: Predicting collective success from individual behavior
• Brain-inspired Search Engine Assistant based on Knowledge Graph
• Adaptive Initialization Method for K-means Algorithm
• Practical Risk Measures in Reinforcement Learning
• NemaNet: A convolutional neural network model for identification of nematodes soybean crop in brazil
• Evaluation of Deep-Learning-Based Voice Activity Detectors and Room Impulse Response Models in Reverberant Environments
• Distributionally Robust Formulation and Model Selection for the Graphical Lasso
• Needmining: Identifying micro blog data containing customer needs
• Detecting Abnormal Profiles in Collaborative Filtering Recommender Systems
• "The Boating Store Had Its Best Sail Ever": Pronunciation-attentive Contextualized Pun Recognition
• On-the-fly Detection of Autogenerated Tweets
• The Atlas for the Aspiring Network Scientist
• A Pattern Recognition Method for Partial Discharge Detection on Insulated Overhead Conductors
• Event Extraction Based on Deep Learning in Food Hazard Arabic Texts
• On Integrating Information Visualization Techniques into Data Mining: A Review
• Semi-Supervised Recognition of the Diploglossus Millepunctatus Lizard Species using Artificial Vision Algorithms
• Dynamic patterns of knowledge flows across technological domains: empirical results and link prediction
• Applying economic measures to lapse risk management with machine learning approaches
• AP-Perf: Incorporating Generic Performance Metrics in Differentiable Learning
• Classification of Breast Cancer Lesions in Ultrasound Images by using Attention Layer and loss Ensembles in Deep Convolutional Neural Networks
• Corner case data description and detection
• The Explanatory Gap in Algorithmic News Curation
• Quantum Pattern Detection: Accurate State- and Circuit-based Analyses
• The OCON model: an old but green solution for distributable supervised classification for acoustic monitoring in smart cities
• Robust Graph Learning Under Wasserstein Uncertainty
• Homonym Detection in Curated Bibliographies: Learning from dblp's Experience (full version)
• Integer-Programming Ensemble of Temporal-Relations Classifiers
• Multi-Level Contextual Network for Biomedical Image Segmentation
• Bayesian analysis of the prevalence bias: learning and predicting from imbalanced data
• Evaluation of Community Detection Methods
• Towards personalized human AI interaction - adapting the behavior of AI agents using neural signatures of subjective interest
• Monitoring the optical quality of the FACT Cherenkov Telescope
• A Process for the Evaluation of Node Embedding Methods in the Context of Node Classification
• Clustering Algorithm for Gujarati Language
• Analysis of Chinese Tourists in Japan by Text Mining of a Hotel Portal Site
• Data-Driven Construction of Data Center Graph of Things for Anomaly Detection
• Subspace Learning in The Presence of Sparse Structured Outliers and Noise
• Enabling Incremental Knowledge Transfer for Object Detection at the Edge
• Predicting Strategic Voting Behavior with Poll Information
• Using RDF Summary Graph For Keyword-based Semantic Searches
• DC-Prophet: Predicting Catastrophic Machine Failures in DataCenters
• Link Prediction Using Supervised Machine Learning based on Aggregated and Topological Features
• Problem Learning: Towards the Free Will of Machines
• Computer Vision Aided URLL Communications: Proactive Service Identification and Coexistence
• A matter of words: NLP for quality evaluation of Wikipedia medical articles
• GeoSay: A Geometric Saliency for Extracting Buildings in Remote Sensing Images
• Classifying movie genres by analyzing text reviews
• Constrained Multi-Objective Optimization for Automated Machine Learning
• Towards Leveraging End-of-Life Tools as an Asset: Value Co-Creation based on Deep Learning in the Machining Industry
• Unsupervised Deep Domain Adaptation for Pedestrian Detection
• Smell Pittsburgh: Community-Empowered Mobile Smell Reporting System
• The Next 700 Policy Miners: A Universal Method for Building Policy Miners
• PanRep: Graph neural networks for extracting universal node embeddings in heterogeneous graphs
• Meta-Learning for One-Class Classification with Few Examples using Order-Equivariant Network
• Relay Selection for 5G New Radio Via Artificial Neural Networks
• DeepClaw: A Robotic Hardware Benchmarking Platform for Learning Object Manipulation
• The Threat to the Validity of Predictive Mutation Testing: The Impact of Uncovered Mutants
• Multiview Learning of Weighted Majority Vote by Bregman Divergence Minimization
• Non-convex non-local flows for saliency detection
• Are there any 'object detectors' in the hidden layers of CNNs trained to identify objects or scenes?
• Image Segmentation Using Subspace Representation and Sparse Decomposition
• Auto-CASH: Autonomous Classification Algorithm Selection with Deep Q-Network
• Inverse Graph Identification: Can We Identify Node Labels Given Graph Labels?
• Teaching a Machine to Diagnose a Heart Disease; Beginning from digitizing scanned ECGs to detecting the Brugada Syndrome (BrS)
• Character Feature Engineering for Japanese Word Segmentation
• Dynamic Embedding on Textual Networks via a Gaussian Process
• Exploratory data analysis for large-scale multiple testing problems and its application in gene expression studies
• Interactive Open-Ended Learning for 3D Object Recognition
• Exploratory studies of human gait changes using depth cameras and considering measurement errors
• PUNCH: Positive UNlabelled Classification based information retrieval in Hyperspectral images
• QRMODA and BRMODA: Novel Models for Face Recognition Accuracy in Computer Vision Systems with Adapted Video Streams
• Text Extraction From Texture Images Using Masked Signal Decomposition
• Progress Estimation and Phase Detection for Sequential Processes
• Classifier Selection with Permutation Tests
• Image Classification of Grapevine Buds using Scale-Invariant Features Transform, Bag of Features and Support Vector Machines
• Estimating Asset Class Health Indices in Power Systems
• Automatic segmentation of lizard spots using an active contour model
• Reducing training requirements through evolutionary based dimension reduction and subject transfer
• Evaluating Protein-protein Interaction Predictors with a Novel 3-Dimensional Metric
• A Semi-Automatic Approach for Detecting Dataset References in Social Science Texts
• Image Segmentation Using Overlapping Group Sparsity
• Defending Grey Attacks by Exploiting Wavelet Analysis in Collaborative Filtering Recommender Systems
• A computationally fast alternative to cross-validation in penalized Gaussian graphical models
• A Harmonic Mean Linear Discriminant Analysis for Robust Image Classification
• ADABOOK & MULTIBOOK: Adaptive Boosting with Chance Correction
• A computationally and cognitively plausible model of supervised and unsupervised learning
• Segmentation of brain tumor on magnetic resonance imaging using a convolutional architecture
• Partially-Supervised Novel Object Captioning Leveraging Context from Paired Data
• Beyond Discriminant Patterns: On the Robustness of Decision Rule Ensembles
• On the Need of Removing Last Releases of Data When Using or Validating Defect Prediction Models
• Time-varying Graph Learning Under Structured Temporal Priors
• Using Statistical and Semantic Models for Multi-Document Summarization
• Predictive Analytics of Varieties of Potatoes
• An Automated Method to Enrich Consumer Health Vocabularies Using GloVe Word Embeddings and An Auxiliary Lexical Resource
• ReadsMap: a new tool for high precision mapping of DNAseq and RNAseq read sequences
• Classification of Contract-Amendment Relationships
• Using Metamorphic Relations to Verify and Enhance Artcode Classification
• Field-Based Plot Extraction Using UAV RGB Images
• CoG: a Two-View Co-training Framework for Defending Adversarial Attacks on Graph
• Relation Analysis between Hotel Review Rating Scores and Sentiment Analysis of Reviews by Chinese Tourists Visiting Japan
• Audio-Visual Embedding for Cross-Modal MusicVideo Retrieval through Supervised Deep CCA
• Estimating sex and age for forensic applications using machine learning based on facial measurements from frontal cephalometric landmarks
• Automatic joint damage quantification using computer vision and deep learning
• Limitation of Acyclic Oriented Graphs Matching as Cell Tracking Accuracy Measure when Evaluating Mitosis
• Fault Prediction based on Software Metrics and SonarQube Rules. Machine or Deep Learning?
• Augmenting expert detection of early coronary artery occlusion from 12 lead electrocardiograms using deep learning
• Static and Dynamic Failure Localization through Progressive Network Tomography
• The Immersion of Directed Multi-graphs in Embedding Fields. Generalisations
• How practical is it? Machine Learning for Identifying Conceptual Interoperability Constraints in API Documents
• Deep Transfer Learning For Plant Center Localization