Putting Fairness Principles into Practice: Challenges, Metrics, and Improvements

Abstract

As more researchers have become aware of and passionate about algorithmic fairness, there has been an explosion in papers laying out new metrics, suggesting algorithms to address issues, and calling attention to issues in existing applications of machine learning. This research has greatly expanded our understanding of the concerns and challenges in deploying machine learning, but there has been much less work in seeing how the rubber meets the road. In this paper we provide a case-study on the application of fairness in machine learning research to a production classification system, and offer new insights in how to measure and address algorithmic fairness issues. We discuss open questions in implementing equality of opportunity and describe our fairness metric, conditional equality, that takes into account distributional differences. Further, we provide a new approach to improve on the fairness metric during model training and demonstrate its efficacy in improving performance for a real-world product

References in corpus (12)

• Equality of Opportunity in Supervised Learning
• Domain Separation Networks
• Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness
• Data Decisions and Theoretical Implications when Adversarially Learning Fair Representations
• On Fairness and Calibration
• Domain-Adversarial Neural Networks
• Learning Adversarially Fair and Transferable Representations
• Satisfying Real-world Goals with Dataset Constraints
• Censoring Representations with an Adversary
• Penalizing Unfairness in Binary Classification
• Calibration for the (Computationally-Identifiable) Masses
• On conditional parity as a notion of non-discrimination in machine learning

Cited by in corpus (5)

• LiFT: A Scalable Framework for Measuring Fairness in ML Applications
• Machine Learning practices and infrastructures
• FairSample: Training Fair and Accurate Graph Convolutional Neural Networks Efficiently
• A non-discriminatory approach to ethical deep learning
• Outlining Traceability: A Principle for Operationalizing Accountability in Computing Systems