Creating Training Sets via Weak Indirect Supervision

Abstract

Creating labeled training sets has become one of the major roadblocks in machine learning. To address this, recent \emph{Weak Supervision (WS)} frameworks synthesize training labels from multiple potentially noisy supervision sources. However, existing frameworks are restricted to supervision sources that share the same output space as the target task. To extend the scope of usable sources, we formulate Weak Indirect Supervision (WIS), a new research problem for automatically synthesizing training labels based on indirect supervision sources that have different output label spaces. To overcome the challenge of mismatched output spaces, we develop a probabilistic modeling approach, PLRM, which uses user-provided label relations to model and leverage indirect supervision sources. Moreover, we provide a theoretically-principled test of the distinguishability of PLRM for unseen labels, along with a generalization bound. On both image and text classification tasks as well as an industrial advertising application, we demonstrate the advantages of PLRM by outperforming baselines by a margin of 2%-9%.

arXiv commentICLR 2022

References in corpus (11)

• Snorkel: Rapid Training Data Creation with Weak Supervision
• LSHTC: A Benchmark for Large-Scale Text Classification
• Learning From Noisy Singly-labeled Data
• Ontology-driven weak supervision for clinical entity classification in electronic health records
• AutoKnow: Self-Driving Knowledge Collection for Products of Thousands of Types
• Generating Multi-Agent Trajectories using Programmatic Weak Supervision
• WRENCH: A Comprehensive Benchmark for Weak Supervision
• Finer Grained Entity Typing with TypeNet
• FrugalML: How to Use ML Prediction APIs More Accurately and Cheaply
• Learning from Indirect Observations
• Estimation from Indirect Supervision with Linear Moments