Harnessing Deep Neural Networks with Logic Rules

Abstract

Combining deep neural networks with structured logic rules is desirable to harness flexibility and reduce uninterpretability of the neural models. We propose a general framework capable of enhancing various types of neural networks (e.g., CNNs and RNNs) with declarative first-order logic rules. Specifically, we develop an iterative distillation method that transfers the structured information of logic rules into the weights of neural networks. We deploy the framework on a CNN for sentiment analysis, and an RNN for named entity recognition. With a few highly intuitive rules, we obtain substantial improvements and achieve state-of-the-art or comparable results to previous best-performing systems.

arXiv commentFix typos in appendix. ACL 2016

References in corpus (11)

• Distributed Representations of Words and Phrases and their Compositionality
• Auto-Encoding Variational Bayes
• Distilling the Knowledge in a Neural Network
• ADADELTA: An Adaptive Learning Rate Method
• Natural Language Processing (almost) from Scratch
• Distributed Representations of Sentences and Documents
• Perceptual Losses for Real-Time Style Transfer and Super-Resolution
• Deep Convolutional Inverse Graphics Network
• Composing graphical models with neural networks for structured representations and fast inference
• Hinge-Loss Markov Random Fields and Probabilistic Soft Logic
• Bayesian representation learning with oracle constraints

Cited by in corpus (86)

• Informed Machine Learning -- A Taxonomy and Survey of Integrating Knowledge into Learning Systems
• A Survey of Knowledge-Enhanced Text Generation
• Syntax-Directed Variational Autoencoder for Structured Data
• Adaptive Multi-Teacher Multi-level Knowledge Distillation
• End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF
• Toward Controlled Generation of Text
• End-to-End Differentiable Proving
• If I Hear You Correctly: Building and Evaluating Interview Chatbots with Active Listening Skills
• On the Origin of Deep Learning
• A Semantic Loss Function for Deep Learning with Symbolic Knowledge
• GraphIE: A Graph-Based Framework for Information Extraction
• Deep Learning-based Sentiment Classification: A Comparative Survey
• On Interpretability of Artificial Neural Networks: A Survey
• Finite Element Network Analysis: A Machine Learning based Computational Framework for the Simulation of Physical Systems
• Visual Interpretability for Deep Learning: a Survey
• TensorLog: Deep Learning Meets Probabilistic DBs
• Neural Compatibility Modeling with Attentive Knowledge Distillation
• Opportunities and Challenges of Deep Learning Methods for Electrocardiogram Data: A Systematic Review
• Knowledge Graph Embedding with Iterative Guidance from Soft Rules
• Interpretable Convolutional Neural Networks
• Unsupervised Learning of Neural Networks to Explain Neural Networks
• Logic Rules Powered Knowledge Graph Embedding
• Knowledge Infused Learning (K-IL): Towards Deep Incorporation of Knowledge in Deep Learning
• Tunneling Neural Perception and Logic Reasoning through Abductive Learning
• Revisit Fuzzy Neural Network: Demystifying Batch Normalization and ReLU with Generalized Hamming Network
• Automatic Article Commenting: the Task and Dataset
• Representation Stability as a Regularizer for Improved Text Analytics Transfer Learning
• Analyzing Differentiable Fuzzy Logic Operators
• Hybrid Predictive Model: When an Interpretable Model Collaborates with a Black-box Model
• Deep Reasoning Networks: Thinking Fast and Slow
• Knowledge Enhanced Attention for Robust Natural Language Inference
• Machine Learning with World Knowledge: The Position and Survey
• Optimizing for Interpretability in Deep Neural Networks with Tree Regularization
• Examining CNN Representations with respect to Dataset Bias
• Neural-encoding Human Experts' Domain Knowledge to Warm Start Reinforcement Learning
• A Roadmap to Domain Knowledge Integration in Machine Learning
• A Curriculum Domain Adaptation Approach to the Semantic Segmentation of Urban Scenes
• Towards a Unified Evaluation of Explanation Methods without Ground Truth
• Integrating Deep Learning with Logic Fusion for Information Extraction
• Rule-Guided Compositional Representation Learning on Knowledge Graphs
• Data-to-Text Generation with Style Imitation
• Deep Adaptive Semantic Logic (DASL): Compiling Declarative Knowledge into Deep Neural Networks
• Data Programming using Continuous and Quality-Guided Labeling Functions
• KoGuN: Accelerating Deep Reinforcement Learning via Integrating Human Suboptimal Knowledge
• Zero-Shot Learning with Knowledge Enhanced Visual Semantic Embeddings
• Coupling Distributed and Symbolic Execution for Natural Language Queries
• Developing Knowledge-enhanced Chronic Disease Risk Prediction Models from Regional EHR Repositories
• Explaining Neural Networks Semantically and Quantitatively
• Automating Crystal-Structure Phase Mapping: Combining Deep Learning with Constraint Reasoning
• A Survey on Neural-symbolic Learning Systems
• Augmenting Neural Networks with First-order Logic
• PK-GCN: Prior Knowledge Assisted Image Classification using Graph Convolution Networks
• Embedding Symbolic Knowledge into Deep Networks
• Approximate Inference in Structured Instances with Noisy Categorical Observations
• Trusted Neural Networks for Safety-Constrained Autonomous Control
• Meta Reinforcement Learning with Autonomous Inference of Subtask Dependencies
• Ontology-based n-ball Concept Embeddings Informing Few-shot Image Classification
• Controlling Neural Networks with Rule Representations
• Interpretable CNNs for Object Classification
• Utilising Deep Learning and Genome Wide Association Studies for Epistatic-Driven Preterm Birth Classification in African-American Women
• Explaining AlphaGo: Interpreting Contextual Effects in Neural Networks
• Model-Agnostic Linear Competitors -- When Interpretable Models Compete and Collaborate with Black-Box Models
• EXplainable Neural-Symbolic Learning (X-NeSyL) methodology to fuse deep learning representations with expert knowledge graphs: the MonuMAI cultural heritage use case
• Churn Reduction via Distillation
• Adversarial Connective-exploiting Networks for Implicit Discourse Relation Classification
• Mining Interpretable AOG Representations from Convolutional Networks via Active Question Answering
• Explanatory Graphs for CNNs
• Integrating Relation Constraints with Neural Relation Extractors
• Neuro-symbolic EDA-based Optimisation using ILP-enhanced DBNs
• Preferential Mixture-of-Experts: Interpretable Models that Rely on Human Expertise as much as Possible
• Marrying up Regular Expressions with Neural Networks: A Case Study for Spoken Language Understanding
• Leveraging Declarative Knowledge in Text and First-Order Logic for Fine-Grained Propaganda Detection
• Structured Prediction in NLP -- A survey
• A Survey on Knowledge integration techniques with Artificial Neural Networks for seq-2-seq/time series models
• Supervised Domain Enablement Attention for Personalized Domain Classification
• Memory networks for consumer protection:unfairness exposed
• An Informal Introduction to Multiplet Neural Networks
• Domain Constraint Approximation based Semi Supervision
• A Base Camp for Scaling AI
• Predicting Customer Call Intent by Analyzing Phone Call Transcripts based on CNN for Multi-Class Classification
• Every time I fire a conversational designer, the performance of the dialog system goes down
• Self-Training with Weak Supervision
• Leveraging Unlabeled Data for Entity-Relation Extraction through Probabilistic Constraint Satisfaction
• Revisiting the Importance of Encoding Logic Rules in Sentiment Classification
• Distantly Supervised Semantic Text Detection and Recognition for Broadcast Sports Videos Understanding
• Understanding in Artificial Intelligence