Fairness Score and Process Standardization: Framework for Fairness Certification in Artificial Intelligence Systems

Abstract

Decisions made by various Artificial Intelligence (AI) systems greatly influence our day-to-day lives. With the increasing use of AI systems, it becomes crucial to know that they are fair, identify the underlying biases in their decision-making, and create a standardized framework to ascertain their fairness. In this paper, we propose a novel Fairness Score to measure the fairness of a data-driven AI system and a Standard Operating Procedure (SOP) for issuing Fairness Certification for such systems. Fairness Score and audit process standardization will ensure quality, reduce ambiguity, enable comparison and improve the trustworthiness of the AI systems. It will also provide a framework to operationalise the concept of fairness and facilitate the commercial deployment of such systems. Furthermore, a Fairness Certificate issued by a designated third-party auditing agency following the standardized process would boost the conviction of the organizations in the AI systems that they intend to deploy. The Bias Index proposed in this paper also reveals comparative bias amongst the various protected attributes within the dataset. To substantiate the proposed framework, we iteratively train a model on biased and unbiased data using multiple datasets and check that the Fairness Score and the proposed process correctly identify the biases and judge the fairness.

arXiv comment15 pages, 4 figures

References in corpus (14)

• Equality of Opportunity in Supervised Learning
• Counterfactual Fairness
• The Measure and Mismeasure of Fairness
• Inherent Trade-Offs in the Fair Determination of Risk Scores
• Bias and Discrimination in AI: a cross-disciplinary perspective
• 50 Years of Test (Un)fairness: Lessons for Machine Learning
• The Frontiers of Fairness in Machine Learning
• CERTIFAI: Counterfactual Explanations for Robustness, Transparency, Interpretability, and Fairness of Artificial Intelligence models
• Achieving Fairness through Adversarial Learning: an Application to Recidivism Prediction
• This Thing Called Fairness: Disciplinary Confusion Realizing a Value in Technology
• On the Moral Justification of Statistical Parity
• Fairness Violations and Mitigation under Covariate Shift
• Data, Power and Bias in Artificial Intelligence
• Fairness-aware machine learning: a perspective

Cited by in corpus (6)

• A five-layer framework for AI governance: integrating regulation, standards, and certification
• (Local) Differential Privacy has NO Disparate Impact on Fairness
• Enhancements for Developing a Comprehensive AI Fairness Assessment Standard
• Nishpaksh: TEC Standard-Compliant Framework for Fairness Auditing and Certification of AI Models
• A federated architecture for sector-led AI governance: lessons from India
• Evaluation of AI Ethics Tools in Language Models: A Developers' Perspective Case Study