A Nutritional Label for Rankings

Abstract

Algorithmic decisions often result in scoring and ranking individuals to determine credit worthiness, qualifications for college admissions and employment, and compatibility as dating partners. While automatic and seemingly objective, ranking algorithms can discriminate against individuals and protected groups, and exhibit low diversity. Furthermore, ranked results are often unstable --- small changes in the input data or in the ranking methodology may lead to drastic changes in the output, making the result uninformative and easy to manipulate. Similar concerns apply in cases where items other than individuals are ranked, including colleges, academic departments, or products. In this demonstration we present Ranking Facts, a Web-based application that generates a "nutritional label" for rankings. Ranking Facts is made up of a collection of visual widgets that implement our latest research results on fairness, stability, and transparency for rankings, and that communicate details of the ranking methodology, or of the output, to the end user. We will showcase Ranking Facts on real datasets from different domains, including college rankings, criminal risk assessment, and financial services.

arXiv comment4 pages, SIGMOD demo, 3 figuress, ACM SIGMOD 2018

References in corpus (1)

• FA*IR: A Fair Top-k Ranking Algorithm

Cited by in corpus (20)

• Prediction-Based Decisions and Fairness: A Catalogue of Choices, Assumptions, and Definitions
• Explainability Fact Sheets: A Framework for Systematic Assessment of Explainable Approaches
• FairSight: Visual Analytics for Fairness in Decision Making
• The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards
• Assessing and Remedying Coverage for a Given Dataset
• On Obtaining Stable Rankings
• Multisource AI Scorecard Table for System Evaluation
• FAT Forensics: A Python Toolbox for Algorithmic Fairness, Accountability and Transparency
• A Framework for Understanding AI-Induced Field Change: How AI Technologies are Legitimized and Institutionalized
• SoK: Machine Learning Governance
• Artificial Intelligence Strategies for National Security and Safety Standards
• Transparency, Fairness, Data Protection, Neutrality: Data Management Challenges in the Face of New Regulation
• Help or Hinder? Evaluating the Impact of Fairness Metrics and Algorithms in Visualizations for Consensus Ranking
• Teaching Responsible Data Science: Charting New Pedagogical Territory
• Technologies for Trustworthy Machine Learning: A Survey in a Socio-Technical Context
• Computational Methods in Professional Communication
• ShaRP: Explaining Rankings and Preferences with Shapley Values
• Completeness of Datasets Documentation on ML/AI repositories: an Empirical Investigation
• Patterns Count-Based Labels for Datasets
• Towards a Flexible Framework for Algorithmic Fairness