WRENCH: A Comprehensive Benchmark for Weak Supervision

Abstract

Recent Weak Supervision (WS) approaches have had widespread success in easing the bottleneck of labeling training data for machine learning by synthesizing labels from multiple potentially noisy supervision sources. However, proper measurement and analysis of these approaches remain a challenge. First, datasets used in existing works are often private and/or custom, limiting standardization. Second, WS datasets with the same name and base data often vary in terms of the labels and weak supervision sources used, a significant "hidden" source of evaluation variance. Finally, WS studies often diverge in terms of the evaluation protocol and ablations used. To address these problems, we introduce a benchmark platform, WRENCH, for thorough and standardized evaluation of WS approaches. It consists of 22 varied real-world datasets for classification and sequence tagging; a range of real, synthetic, and procedurally-generated weak supervision sources; and a modular, extensible framework for WS evaluation, including implementations for popular WS methods. We use WRENCH to conduct extensive comparisons over more than 120 method variants to demonstrate its efficacy as a benchmark platform. The code is available at https://github.com/JieyuZ2/wrench.

NeurIPS 2021 Datasets and Benchmarks Track

References in corpus (16)

• Snorkel: Rapid Training Data Creation with Weak Supervision
• SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations Between Pairs of Nominals
• Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels
• Learning the Structure of Generative Models without Labeled Data
• Learning with Noise: Enhance Distantly Supervised Relation Extraction with Dynamic Transition Matrix
• Ontology-driven weak supervision for clinical entity classification in electronic health records
• SwellShark: A Generative Model for Biomedical Named Entity Recognition without Labeled Data
• Denoising Multi-Source Weak Supervision for Neural Text Classification
• Learning from Rules Generalizing Labeled Exemplars
• GOGGLES: Automatic Image Labeling with Affinity Coding
• Rekall: Specifying Video Events using Compositions of Spatiotemporal Labels
• PPR-FCN: Weakly Supervised Visual Relation Detection via Parallel Pairwise R-FCN
• Heterogeneous Supervision for Relation Extraction: A Representation Learning Approach
• QUEACO: Borrowing Treasures from Weakly-labeled Behavior Data for Query Attribute Value Extraction
• Distantly Supervised Named Entity Recognition using Positive-Unlabeled Learning
• Dependency Structure Misspecification in Multi-Source Weak Supervision Models

Cited by in corpus (10)

• Weakly-Supervised Scientific Document Classification via Retrieval-Augmented Multi-Stage Training
• Sparse Conditional Hidden Markov Model for Weakly Supervised Named Entity Recognition
• Adaptive Multi-view Rule Discovery for Weakly-Supervised Compatible Products Prediction
• Weakly Supervised Label Learning Flows
• Local Boosting for Weakly-Supervised Learning
• Creating Training Sets via Weak Indirect Supervision
• Learning with Noisy Labels by Adaptive Gradient-Based Outlier Removal
• ULF: Unsupervised Labeling Function Correction using Cross-Validation for Weak Supervision
• Neural-Hidden-CRF: A Robust Weakly-Supervised Sequence Labeler
• Characterizing Online Criticism of Partisan News Media Using Weakly Supervised Learning