On the relation between accuracy and fairness in binary classification

Abstract

Our study revisits the problem of accuracy-fairness tradeoff in binary classification. We argue that comparison of non-discriminatory classifiers needs to account for different rates of positive predictions, otherwise conclusions about performance may be misleading, because accuracy and discrimination of naive baselines on the same dataset vary with different rates of positive predictions. We provide methodological recommendations for sound comparison of non-discriminatory classifiers, and present a brief theoretical and empirical analysis of tradeoffs between accuracy and non-discrimination.

Accepted for presentation to the 2nd workshop on Fairness, Accountability, and Transparency in Machine Learning (http://www.fatml.org/)

Cited by in corpus (35)

• Equality of Opportunity in Supervised Learning
• Fairness of Exposure in Rankings
• Fairness in Machine Learning: A Survey
• Weakly-Supervised Disentanglement Without Compromises
• A survey on measuring indirect discrimination in machine learning
• Empirical Risk Minimization under Fairness Constraints
• Aequitas: A Bias and Fairness Audit Toolkit
• Machine learning fairness notions: Bridging the gap with real-world applications
• Policy Learning for Fairness in Ranking
• A survey on datasets for fairness-aware machine learning
• Assessing Algorithmic Fairness with Unobserved Protected Class Using Data Combination
• Inherent Tradeoffs in Learning Fair Representations
• Leveraging Labeled and Unlabeled Data for Consistent Fair Binary Classification
• FACT: A Diagnostic for Group Fairness Trade-offs
• Evaluation of group fairness measures in student performance prediction problems
• Review of Mathematical frameworks for Fairness in Machine Learning
• The Authority of "Fair" in Machine Learning
• Fairness-aware machine learning: a perspective
• Random Hypervolume Scalarizations for Provable Multi-Objective Black Box Optimization
• Evaluating Fairness Metrics in the Presence of Dataset Bias
• Spectral Algorithms for Computing Fair Support Vector Machines
• Disentangled Representations from Non-Disentangled Models
• On the Fairness of Disentangled Representations
• Fairness in Forecasting and Learning Linear Dynamical Systems
• Learning Fair Rule Lists
• Fairness in Forecasting of Observations of Linear Dynamical Systems
• Dynamic Modeling and Equilibria in Fair Decision Making
• No-regret Algorithms for Multi-task Bayesian Optimization
• Technologies for Trustworthy Machine Learning: A Survey in a Socio-Technical Context
• Fairness in Ranking under Uncertainty
• Balancing Competing Objectives with Noisy Data: Score-Based Classifiers for Welfare-Aware Machine Learning
• Fair Sparse Regression with Clustering: An Invex Relaxation for a Combinatorial Problem
• Evaluating Fairness of Machine Learning Models Under Uncertain and Incomplete Information
• Characterizing the risk of fairwashing
• Coping with Mistreatment in Fair Algorithms