Human Factors in Model Interpretability: Industry Practices, Challenges, and Needs

Abstract

As the use of machine learning (ML) models in product development and data-driven decision-making processes became pervasive in many domains, people's focus on building a well-performing model has increasingly shifted to understanding how their model works. While scholarly interest in model interpretability has grown rapidly in research communities like HCI, ML, and beyond, little is known about how practitioners perceive and aim to provide interpretability in the context of their existing workflows. This lack of understanding of interpretability as practiced may prevent interpretability research from addressing important needs, or lead to unrealistic solutions. To bridge this gap, we conducted 22 semi-structured interviews with industry practitioners to understand how they conceive of and design for interpretability while they plan, build, and use their models. Based on a qualitative analysis of our results, we differentiate interpretability roles, processes, goals and strategies as they exist within organizations making heavy use of ML models. The characterization of interpretability work that emerges from our analysis suggests that model interpretability frequently involves cooperation and mental model comparison between people in different roles, often aimed at building trust not only between people and models but also between people within the organization. We present implications for design that discuss gaps between the interpretability challenges that practitioners face in their practice and approaches proposed in the literature, highlighting possible research directions that can better address real-world needs.

ACM CSCW 2020

References in corpus (13)

• Towards A Rigorous Science of Interpretable Machine Learning
• Interpretable classifiers using rules and Bayesian analysis: Building a better stroke prediction model
• Improving fairness in machine learning systems: What do industry practitioners need?
• Human-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AI
• Manifold: A Model-Agnostic Framework for Interpretation and Diagnosis of Machine Learning Models
• How do Humans Understand Explanations from Machine Learning Systems? An Evaluation of the Human-Interpretability of Explanation
• Direct-Manipulation Visualization of Deep Networks
• Quantifying Interpretability and Trust in Machine Learning Systems
• Dissonance Between Human and Machine Understanding
• Human-in-the-Loop Interpretability Prior
• BayesDB: A probabilistic programming system for querying the probable implications of data
• Towards Explainable Deep Learning for Credit Lending: A Case Study
• The Effect of Heterogeneous Data for Alzheimer's Disease Detection from Speech

Cited by in corpus (30)

• Expanding Explainability: Towards Social Transparency in AI systems
• "Help Me Help the AI": Understanding How Explainability Can Support Human-AI Interaction
• Beyond Expertise and Roles: A Framework to Characterize the Stakeholders of Interpretable Machine Learning and their Needs
• Interactive Model Cards: A Human-Centered Approach to Model Documentation
• Sensible AI: Re-imagining Interpretability and Explainability using Sensemaking Theory
• Explainable Machine Learning for Public Policy: Use Cases, Gaps, and Research Directions
• Farsight: Fostering Responsible AI Awareness During AI Application Prototyping
• Fairness via Explanation Quality: Evaluating Disparities in the Quality of Post hoc Explanations
• Machine Learning Practices Outside Big Tech: How Resource Constraints Challenge Responsible Development
• Seamful XAI: Operationalizing Seamful Design in Explainable AI
• Knowledge Graphs in Practice: Characterizing their Users, Challenges, and Visualization Opportunities
• Epistemic values in feature importance methods: Lessons from feminist epistemology
• FeatureEnVi: Visual Analytics for Feature Engineering Using Stepwise Selection and Semi-Automatic Extraction Approaches
• RES: A Robust Framework for Guiding Visual Explanation
• Artificial Concepts of Artificial Intelligence: Institutional Compliance and Resistance in AI Startups
• Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning Experiences
• Interpretability, Then What? Editing Machine Learning Models to Reflect Human Knowledge and Values
• Explanation Strategies as an Empirical-Analytical Lens for Socio-Technical Contextualization of Machine Learning Interpretability
• Human-Centered Explainable AI (XAI): From Algorithms to User Experiences
• 3DPFIX: Improving Remote Novices' 3D Printing Troubleshooting through Human-AI Collaboration
• Interpretable Machine Learning: Moving From Mythos to Diagnostics
• Are Metrics Enough? Guidelines for Communicating and Visualizing Predictive Models to Subject Matter Experts
• Identifying Explanation Needs of End-users: Applying and Extending the XAI Question Bank
• "Is It My Turn?" Assessing Teamwork and Taskwork in Collaborative Immersive Analytics
• "It answers questions that I didn't know I had": Ph.D. Students' Evaluation of an Information Sharing Knowledge Graph
• Modeling Disclosive Transparency in NLP Application Descriptions
• Whither AutoML? Understanding the Role of Automation in Machine Learning Workflows
• VibE: A Visual Analytics Workflow for Semantic Error Analysis of CVML Models at Subgroup Level
• Towards Evaluating Exploratory Model Building Process with AutoML Systems
• Hierarchical Decision Ensembles- An inferential framework for uncertain Human-AI collaboration in forensic examinations