Model Sketching: Centering Concepts in Early-Stage Machine Learning Model Design
arXiv:2303.02884 · doi:10.1145/3544548.3581290
Abstract
Machine learning practitioners often end up tunneling on low-level technical details like model architectures and performance metrics. Could early model development instead focus on high-level questions of which factors a model ought to pay attention to? Inspired by the practice of sketching in design, which distills ideas to their minimal representation, we introduce model sketching: a technical framework for iteratively and rapidly authoring functional approximations of a machine learning model's decision-making logic. Model sketching refocuses practitioner attention on composing high-level, human-understandable concepts that the model is expected to reason over (e.g., profanity, racism, or sarcasm in a content moderation task) using zero-shot concept instantiation. In an evaluation with 17 ML practitioners, model sketching reframed thinking from implementation to higher-level exploration, prompted iteration on a broader range of model designs, and helped identify gaps in the problem formulation$\unicode{x2014}$all in a fraction of the time ordinarily required to build a model.
arXiv commentTo appear at CHI 2023
References in corpus (12)
- Scikit-learn: Machine Learning in Python
- Human-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AI
- Measurement and Fairness
- Problem Formulation and Fairness
- Trust in Data Science: Collaboration, Translation, and Accountability in Corporate Data Science Projects
- Concept Bottleneck Models
- How AI Developers Overcome Communication Challenges in a Multidisciplinary Team: A Case Study
- Jury Learning: Integrating Dissenting Voices into Machine Learning Models
- How Data Scientists Work Together With Domain Experts in Scientific Collaborations: To Find The Right Answer Or To Ask The Right Question?
- Data Vision: Learning to See Through Algorithmic Abstraction
- Enhance Multimodal Transformer With External Label And In-Domain Pretrain: Hateful Meme Challenge Winning Solution
- User Modelling for Avoiding Overfitting in Interactive Knowledge Elicitation for Prediction
Cited by in corpus (9)
- ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing
- Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooM
- Sketching AI Concepts with Capabilities and Examples: AI Innovation in the Intensive Care Unit
- SuperNOVA: Design Strategies and Opportunities for Interactive Visualization in Computational Notebooks
- Making the Right Thing: Bridging HCI and Responsible AI in Early-Stage AI Concept Selection
- Exploring the Innovation Opportunities for Pre-trained Models
- Clarify: Improving Model Robustness With Natural Language Corrections
- Policy Maps: Tools for Guiding the Unbounded Space of LLM Behaviors
- Tempo: Helping Data Scientists and Domain Experts Collaboratively Specify Predictive Modeling Tasks