Completeness of Datasets Documentation on ML/AI repositories: an Empirical Investigation

Abstract

ML/AI is the field of computer science and computer engineering that arguably received the most attention and funding over the last decade. Data is the key element of ML/AI, so it is becoming increasingly important to ensure that users are fully aware of the quality of the datasets that they use, and of the process generating them, so that possible negative impacts on downstream effects can be tracked, analysed, and, where possible, mitigated. One of the tools that can be useful in this perspective is dataset documentation. The aim of this work is to investigate the state of dataset documentation practices, measuring the completeness of the documentation of several popular datasets in ML/AI repositories. We created a dataset documentation schema -- the Documentation Test Sheet (DTS) -- that identifies the information that should always be attached to a dataset (to ensure proper dataset choice and informed use), according to relevant studies in the literature. We verified 100 popular datasets from four different repositories with the DTS to investigate which information was present. Overall, we observed a lack of relevant documentation, especially about the context of data collection and data processing, highlighting a paucity of transparency.

References in corpus (5)

• Model Cards for Model Reporting
• Lessons from Archives: Strategies for Collecting Sociocultural Data in Machine Learning
• Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development
• A Nutritional Label for Rankings
• Augmented Datasheets for Speech Datasets and Ethical Decision-Making