ConfusionFlow: A model-agnostic visualization for temporal analysis of classifier confusion

Abstract

Classifiers are among the most widely used supervised machine learning algorithms. Many classification models exist, and choosing the right one for a given task is difficult. During model selection and debugging, data scientists need to assess classifiers' performances, evaluate their learning behavior over time, and compare different models. Typically, this analysis is based on single-number performance measures such as accuracy. A more detailed evaluation of classifiers is possible by inspecting class errors. The confusion matrix is an established way for visualizing these class errors, but it was not designed with temporal or comparative analysis in mind. More generally, established performance analysis systems do not allow a combined temporal and comparative analysis of class-level information. To address this issue, we propose ConfusionFlow, an interactive, comparative visualization tool that combines the benefits of class confusion matrices with the visualization of performance characteristics over time. ConfusionFlow is model-agnostic and can be used to compare performances for different model types, model architectures, and/or training and test datasets. We demonstrate the usefulness of ConfusionFlow in a case study on instance selection strategies in active learning. We further assess the scalability of ConfusionFlow and present a use case in the context of neural network pruning.

Changes compared to previous version: Reintroduced NN pruning use case; restructured Evaluation section; several additional minor revisions. Submitted as Minor Revision to IEEE TVCG on 2020-07-02

References in corpus (14)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• OpenML: networked science in machine learning
• Rethinking the Value of Network Pruning
• The What-If Tool: Interactive Probing of Machine Learning Models
• Manifold: A Model-Agnostic Framework for Interpretation and Diagnosis of Machine Learning Models
• GAN Lab: Understanding Complex Deep Generative Models using Interactive Visual Experimentation
• Do Convolutional Neural Networks Learn Class Hierarchy?
• Do CIFAR-10 Classifiers Generalize to CIFAR-10?
• Direct-Manipulation Visualization of Deep Networks
• CNNComparator: Comparative Analytics of Convolutional Neural Networks
• DeepTracker: Visualizing the Training Process of Convolutional Neural Networks
• InstanceFlow: Visualizing the Evolution of Classifier Confusion on the Instance Level

Cited by in corpus (8)

• Neo: Generalizing Confusion Matrix Visualization to Hierarchical and Multi-Output Labels
• Zeno: An Interactive Framework for Behavioral Evaluation of Machine Learning
• Symphony: Composing Interactive Interfaces for Machine Learning
• InstanceFlow: Visualizing the Evolution of Classifier Confusion on the Instance Level
• RouteExplainer: An Explanation Framework for Vehicle Routing Problem
• Model-agnostic interpretation by visualization of feature perturbations
• VERB: Visualizing and Interpreting Bias Mitigation Techniques for Word Representations
• IKIWISI: An Interactive Visual Pattern Generator for Evaluating the Reliability of Vision-Language Models Without Ground Truth