Training Set Debugging Using Trusted Items

Abstract

Training set bugs are flaws in the data that adversely affect machine learning. The training set is usually too large for man- ual inspection, but one may have the resources to verify a few trusted items. The set of trusted items may not by itself be adequate for learning, so we propose an algorithm that uses these items to identify bugs in the training set and thus im- proves learning. Specifically, our approach seeks the smallest set of changes to the training set labels such that the model learned from this corrected training set predicts labels of the trusted items correctly. We flag the items whose labels are changed as potential bugs, whose labels can be checked for veracity by human experts. To find the bugs in this way is a challenging combinatorial bilevel optimization problem, but it can be relaxed into a continuous optimization problem. Ex- periments on toy and real data demonstrate that our approach can identify training set bugs effectively and suggest appro- priate changes to the labels. Our algorithm is a step toward trustworthy machine learning.

AAAI 2018

Cited by in corpus (12)

• Data Cleansing for Models Trained with SGD
• FR-Train: A Mutual Information-Based Approach to Fair and Robust Training
• Defense Against Reward Poisoning Attacks in Reinforcement Learning
• Regularisation Can Mitigate Poisoning Attacks: A Novel Analysis Based on Multiobjective Bilevel Optimisation
• Regularization Can Help Mitigate Poisoning Attacks... with the Right Hyperparameters
• Efficient Estimation of Influence of a Training Instance
• Reliable Weakly Supervised Learning: Maximize Gain and Maintain Safeness
• Iterative Teaching by Label Synthesis
• De-Pois: An Attack-Agnostic Defense against Data Poisoning Attacks
• Defuse: Harnessing Unrestricted Adversarial Examples for Debugging Models Beyond Test Accuracy
• Model-Agnostic Explanations using Minimal Forcing Subsets
• Provable Training Set Debugging for Linear Regression