Model Sketching: Centering Concepts in Early-Stage Machine Learning Model Design

Abstract

Machine learning practitioners often end up tunneling on low-level technical details like model architectures and performance metrics. Could early model development instead focus on high-level questions of which factors a model ought to pay attention to? Inspired by the practice of sketching in design, which distills ideas to their minimal representation, we introduce model sketching: a technical framework for iteratively and rapidly authoring functional approximations of a machine learning model's decision-making logic. Model sketching refocuses practitioner attention on composing high-level, human-understandable concepts that the model is expected to reason over (e.g., profanity, racism, or sarcasm in a content moderation task) using zero-shot concept instantiation. In an evaluation with 17 ML practitioners, model sketching reframed thinking from implementation to higher-level exploration, prompted iteration on a broader range of model designs, and helped identify gaps in the problem formulation$\unicode{x2014}$all in a fraction of the time ordinarily required to build a model.

To appear at CHI 2023

References in corpus (6)

• Problem Formulation and Fairness
• Trust in Data Science: Collaboration, Translation, and Accountability in Corporate Data Science Projects
• How AI Developers Overcome Communication Challenges in a Multidisciplinary Team: A Case Study
• Jury Learning: Integrating Dissenting Voices into Machine Learning Models
• Data Vision: Learning to See Through Algorithmic Abstraction
• Enhance Multimodal Transformer With External Label And In-Domain Pretrain: Hateful Meme Challenge Winning Solution