BoostClean: Automated Error Detection and Repair for Machine Learning

Abstract

Predictive models based on machine learning can be highly sensitive to data error. Training data are often combined with a variety of different sources, each susceptible to different types of inconsistencies, and new data streams during prediction time, the model may encounter previously unseen inconsistencies. An important class of such inconsistencies is domain value violations that occur when an attribute value is outside of an allowed domain. We explore automatically detecting and repairing such violations by leveraging the often available clean test labels to determine whether a given detection and repair combination will improve model accuracy. We present BoostClean which automatically selects an ensemble of error detection and repair combinations using statistical boosting. BoostClean selects this ensemble from an extensible library that is pre-populated general detection functions, including a novel detector based on the Word2Vec deep learning model, which detects errors across a diverse set of domains. Our evaluation on a collection of 12 datasets from Kaggle, the UCI repository, real-world data analyses, and production datasets that show that Boost- Clean can increase absolute prediction accuracy by up to 9% over the best non-ensembled alternatives. Our optimizations including parallelism, materialization, and indexing techniques show a 22.2x end-to-end speedup on a 16-core machine.

References in corpus (3)

• HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
• The Missing Piece in Complex Analytics: Low Latency, Scalable Model Management and Serving with Velox
• The MADlib Analytics Library or MAD Skills, the SQL

Cited by in corpus (16)

• AutoML: A Survey of the State-of-the-Art
• A Survey on Data Collection for Machine Learning: a Big Data -- AI Integration Perspective
• Automated data processing and feature engineering for deep learning and big data applications: a survey
• Machine Learning Testing: Survey, Landscapes and Horizons
• DiffPrep: Differentiable Data Preprocessing Pipeline Search for Learning over Tabular Data
• Revealing the Semantics of Data Wrangling Scripts With COMANTICS
• Nearest Neighbor Classifiers over Incomplete Information: From Certain Answers to Certain Predictions
• ShapeFlow: Dynamic Shape Interpreter for TensorFlow
• AutoCure: Automated Tabular Data Curation Technique for ML Pipelines
• Data Curation and Quality Assurance for Machine Learning-based Cyber Intrusion Detection
• A Unified Framework for Task-Driven Data Quality Management
• Enabling SQL-based Training Data Debugging for Federated Learning
• ED2: Two-stage Active Learning for Error Detection -- Technical Report
• Isolating Noisy Labelled Test Cases in Human-in-the-Loop Oracle Learning
• Man versus Machine: AutoML and Human Experts' Role in Phishing Detection
• Validate and Enable Machine Learning in Industrial AI