Zeno: An Interactive Framework for Behavioral Evaluation of Machine Learning

Abstract

Machine learning models with high accuracy on test data can still produce systematic failures, such as harmful biases and safety issues, when deployed in the real world. To detect and mitigate such failures, practitioners run behavioral evaluation of their models, checking model outputs for specific types of inputs. Behavioral evaluation is important but challenging, requiring that practitioners discover real-world patterns and validate systematic failures. We conducted 18 semi-structured interviews with ML practitioners to better understand the challenges of behavioral evaluation and found that it is a collaborative, use-case-first process that is not adequately supported by existing task- and domain-specific tools. Using these findings, we designed Zeno, a general-purpose framework for visualizing and testing AI systems across diverse use cases. In four case studies with participants using Zeno on real-world models, we found that practitioners were able to reproduce previous manual analyses and discover new systematic failures.

References in corpus (19)

• On the Opportunities and Risks of Foundation Models
• Model Cards for Model Reporting
• The What-If Tool: Interactive Probing of Machine Learning Models
• Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
• Mobile Sensor Data Anonymization
• Predictive Inequity in Object Detection
• Interactive Model Cards: A Human-Centered Approach to Model Documentation
• Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits
• Neo: Generalizing Confusion Matrix Visualization to Hierarchical and Multi-Output Labels
• Discovering and Validating AI Errors With Crowdsourced Failure Reports
• ConfusionFlow: A model-agnostic visualization for temporal analysis of classifier confusion
• Symphony: Composing Interactive Interfaces for Machine Learning
• Machine Learning Practices Outside Big Tech: How Resource Constraints Challenge Responsible Development
• Domino: Discovering Systematic Errors with Cross-Modal Embeddings
• Slice-based Learning: A Programming Model for Residual Learning in Critical Data Slices
• Collaboration Challenges in Building ML-Enabled Systems: Communication, Documentation, Engineering, and Process
• Did the Model Change? Efficiently Assessing Machine Learning API Shifts
• DendroMap: Visual Exploration of Large-Scale Image Datasets for Machine Learning with Treemaps
• Capabilities for Better ML Engineering

Cited by in corpus (14)

• EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria
• Interactive Debugging and Steering of Multi-Agent AI Systems
• Visualization for Trust in Machine Learning Revisited: The State of the Field in 2023
• Compress and Compare: Interactively Evaluating Efficiency and Behavior Across ML Model Compression Experiments
• Where Does My Model Underperform? A Human Evaluation of Slice Discovery Algorithms
• (Beyond) Reasonable Doubt: Challenges that Public Defenders Face in Scrutinizing AI in Court
• Talaria: Interactively Optimizing Machine Learning Models for Efficient Inference
• Abstraction Alignment: Comparing Model-Learned and Human-Encoded Conceptual Relationships
• Divisi: Interactive Search and Visualization for Scalable Exploratory Subgroup Analysis
• Clarify: Improving Model Robustness With Natural Language Corrections
• VibE: A Visual Analytics Workflow for Semantic Error Analysis of CVML Models at Subgroup Level
• Policy Maps: Tools for Guiding the Unbounded Space of LLM Behaviors
• Tempo: Helping Data Scientists and Domain Experts Collaboratively Specify Predictive Modeling Tasks
• Evaluation of AI Ethics Tools in Language Models: A Developers' Perspective Case Study