Angler: Helping Machine Translation Practitioners Prioritize Model Improvements

Abstract

Machine learning (ML) models can fail in unexpected ways in the real world, but not all model failures are equal. With finite time and resources, ML practitioners are forced to prioritize their model debugging and improvement efforts. Through interviews with 13 ML practitioners at Apple, we found that practitioners construct small targeted test sets to estimate an error's nature, scope, and impact on users. We built on this insight in a case study with machine translation models, and developed Angler, an interactive visual analytics tool to help practitioners prioritize model improvements. In a user study with 7 machine translation experts, we used Angler to understand prioritization practices when the input space is infinite, and obtaining reliable signals of model quality is expensive. Our study revealed that participants could form more interesting and user-focused hypotheses for prioritization by analyzing quantitative summary statistics and qualitatively assessing data by reading sentences.

arXiv commentAccepted to CHI 2023. 20 pages, 6 figures

References in corpus (18)

• Scikit-learn: Machine Learning in Python
• UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
• BERTopic: Neural topic modeling with a class-based TF-IDF procedure
• Improving fairness in machine learning systems: What do industry practitioners need?
• The What-If Tool: Interactive Probing of Machine Learning Models
• Do ImageNet Classifiers Generalize to ImageNet?
• WILDS: A Benchmark of in-the-Wild Distribution Shifts
• Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation
• explAIner: A Visual Analytics Framework for Interactive and Explainable Machine Learning
• Everyday algorithm auditing: Understanding the power of everyday users in surfacing harmful algorithmic behaviors
• Does Object Recognition Work for Everyone?
• Discovering and Validating AI Errors With Crowdsourced Failure Reports
• Meta-Evaluation of Translation Evaluation Methods: a systematic up-to-date overview
• A Large Parallel Corpus of Full-Text Scientific Articles
• Extending Challenge Sets to Uncover Gender Bias in Machine Translation: Impact of Stereotypical Verbs and Adjectives
• UDIS: Unsupervised Discovery of Bias in Deep Visual Recognition Models
• Fair SA: Sensitivity Analysis for Fairness in Face Recognition
• Designing Data: Proactive Data Collection and Iteration for Machine Learning

Cited by in corpus (6)

• EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria
• Farsight: Fostering Responsible AI Awareness During AI Application Prototyping
• Compress and Compare: Interactively Evaluating Efficiency and Behavior Across ML Model Compression Experiments
• RAGTrace: Understanding and Refining Retrieval-Generation Dynamics in Retrieval-Augmented Generation
• Divisi: Interactive Search and Visualization for Scalable Exploratory Subgroup Analysis
• Abstraction Alignment: Comparing Model-Learned and Human-Encoded Conceptual Relationships