Interactive Label Cleaning with Example-based Explanations

Abstract

We tackle sequential learning under label noise in applications where a human supervisor can be queried to relabel suspicious examples. Existing approaches are flawed, in that they only relabel incoming examples that look "suspicious" to the model. As a consequence, those mislabeled examples that elude (or don't undergo) this cleaning step end up tainting the training data and the model with no further chance of being cleaned. We propose Cincer, a novel approach that cleans both new and past data by identifying pairs of mutually incompatible examples. Whenever it detects a suspicious example, Cincer identifies a counter-example in the training set that -- according to the model -- is maximally incompatible with the suspicious example, and asks the annotator to relabel either or both examples, resolving this possible inconsistency. The counter-examples are chosen to be maximally incompatible, so to serve as explanations of the model's suspicion, and highly influential, so to convey as much information as possible if relabeled. Cincer achieves this by leveraging an efficient and robust approximation of influence functions based on the Fisher information matrix (FIM). Our extensive empirical evaluation shows that clarifying the reasons behind the model's suspicions by cleaning the counter-examples helps in acquiring substantially better data and models, especially when paired with our FIM approximation.

arXiv commentmain article + supplementary material, Advances in Neural Information Processing Systems 34 (NeurIPS 2021)

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Understanding Black-box Predictions via Influence Functions
• Optimizing Neural Networks with Kronecker-factored Approximate Curvature
• On the Accuracy of Influence Functions for Measuring Group Effects
• Representer Point Selection for Explaining Deep Neural Networks
• Interpreting Black Box Predictions using Fisher Kernels
• Limitations of the Empirical Fisher Approximation for Natural Gradient Descent
• "Influence Sketching": Finding Influential Samples In Large-Scale Regressions
• Influence Functions in Deep Learning Are Fragile
• Machine Guides, Human Supervises: Interactive Learning with Global Explanations

Cited by in corpus (1)

• Toward a Unified Framework for Debugging Concept-based Models