Deep Probabilistic Logic: A Unifying Framework for Indirect Supervision

Abstract

Deep learning has emerged as a versatile tool for a wide range of NLP tasks, due to its superior capacity in representation learning. But its applicability is limited by the reliance on annotated examples, which are difficult to produce at scale. Indirect supervision has emerged as a promising direction to address this bottleneck, either by introducing labeling functions to automatically generate noisy examples from unlabeled text, or by imposing constraints over interdependent label decisions. A plethora of methods have been proposed, each with respective strengths and limitations. Probabilistic logic offers a unifying language to represent indirect supervision, but end-to-end modeling with probabilistic logic is often infeasible due to intractable inference and learning. In this paper, we propose deep probabilistic logic (DPL) as a general framework for indirect supervision, by composing probabilistic logic with deep learning. DPL models label decisions as latent variables, represents prior knowledge on their relations using weighted first-order logical formulas, and alternates between learning a deep neural network for the end task and refining uncertain formula weights for indirect supervision, using variational EM. This framework subsumes prior indirect supervision methods as special cases, and enables novel combination via infusion of rich domain and linguistic knowledge. Experiments on biomedical machine reading demonstrate the promise of this approach.

arXiv commentEMNLP 2018 final version

References in corpus (7)

• Sequence to Sequence Learning with Neural Networks
• Teaching Machines to Read and Comprehend
• Loopy Belief Propagation for Approximate Inference: An Empirical Study
• Composing graphical models with neural networks for structured representations and fast inference
• Learning the Structure of Generative Models without Labeled Data
• Leveraging Deep Neural Networks and Knowledge Graphs for Entity Disambiguation
• Heterogeneous Supervision for Relation Extraction: A Representation Learning Approach

Cited by in corpus (8)

• Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
• CheXpert++: Approximating the CheXpert labeler for Speed,Differentiability, and Probabilistic Output
• Integrating Deep Learning with Logic Fusion for Information Extraction
• Prior Adaptive Semi-supervised Learning with Application to EHR Phenotyping
• Learning to Denoise Distantly-Labeled Data for Entity Typing
• Augmenting Neural Networks with First-order Logic
• Extreme Multi-label Classification from Aggregated Labels
• Leveraging Declarative Knowledge in Text and First-Order Logic for Fine-Grained Propaganda Detection