Aspirations and Practice of Model Documentation: Moving the Needle with Nudging and Traceability

Abstract

The documentation practice for machine-learned (ML) models often falls short of established practices for traditional software, which impedes model accountability and inadvertently abets inappropriate or misuse of models. Recently, model cards, a proposal for model documentation, have attracted notable attention, but their impact on the actual practice is unclear. In this work, we systematically study the model documentation in the field and investigate how to encourage more responsible and accountable documentation practice. Our analysis of publicly available model cards reveals a substantial gap between the proposal and the practice. We then design a tool named DocML aiming to (1) nudge the data scientists to comply with the model cards proposal during the model development, especially the sections related to ethics, and (2) assess and manage the documentation quality. A lab study reveals the benefit of our tool towards long-term documentation quality and accountability.

To be published in proceedings of CHI 2023

References in corpus (5)

• Improving fairness in machine learning systems: What do industry practitioners need?
• Interactive Model Cards: A Human-Centered Approach to Model Documentation
• Documentation Matters: Human-Centered AI System to Assist Data Science Code Documentation in Computational Notebooks
• CrossData: Leveraging Text-Data Connections for Authoring Data Documents
• Machine Learning Practices Outside Big Tech: How Resource Constraints Challenge Responsible Development

Cited by in corpus (6)

• SuperNOVA: Design Strategies and Opportunities for Interactive Visualization in Computational Notebooks
• Access Denied: Meaningful Data Access for Quantitative Algorithm Audits
• The State of Documentation Practices of Third-party Machine Learning Models and Datasets
• Model Cards Revisited: Bridging the Gap Between Theory and Practice for Ethical AI Requirements
• Talking About the Assumption in the Room
• MRM3: Machine Readable ML Model Metadata