Multi-Hop Knowledge Graph Reasoning with Reward Shaping

Abstract

Multi-hop reasoning is an effective approach for query answering (QA) over incomplete knowledge graphs (KGs). The problem can be formulated in a reinforcement learning (RL) setup, where a policy-based agent sequentially extends its inference path until it reaches a target. However, in an incomplete KG environment, the agent receives low-quality rewards corrupted by false negatives in the training data, which harms generalization at test time. Furthermore, since no golden action sequence is used for training, the agent can be misled by spurious search trajectories that incidentally lead to the correct answer. We propose two modeling advances to address both issues: (1) we reduce the impact of false negative supervision by adopting a pretrained one-hop embedding model to estimate the reward of unobserved facts; (2) we counter the sensitivity to spurious paths of on-policy RL by forcing the agent to explore a diverse set of paths using randomly generated edge masks. Our approach significantly improves over existing path-based KGQA models on several benchmark datasets and is comparable or better than embedding-based models.

arXiv commentAccepted to EMNLP 2018, 12 pages

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Playing Atari with Deep Reinforcement Learning
• A Deep Reinforced Model for Abstractive Summarization
• Complex Embeddings for Simple Link Prediction
• Sequence Level Training with Recurrent Neural Networks
• Go for a Walk and Arrive at the Answer: Reasoning Over Paths in Knowledge Bases using Reinforcement Learning
• Differentiable Learning of Logical Rules for Knowledge Base Reasoning
• End-to-End Differentiable Proving
• DeepPath: A Reinforcement Learning Method for Knowledge Graph Reasoning
• From Language to Programs: Bridging Reinforcement Learning and Maximum Marginal Likelihood
• Learning Symmetric Collaborative Dialogue Agents with Dynamic Knowledge Graph Embeddings
• Variational Knowledge Graph Reasoning

Cited by in corpus (19)

• Transfer Learning in Deep Reinforcement Learning: A Survey
• Do Transformers Really Perform Bad for Graph Representation?
• CAFE: Coarse-to-Fine Neural Symbolic Reasoning for Explainable Recommendation
• Reinforcement Learning from Reformulations in Conversational Question Answering over Knowledge Graphs
• xERTE: Explainable Reasoning on Temporal Knowledge Graphs for Forecasting Future Links
• Dynamic Anticipation and Completion for Multi-Hop Reasoning over Sparse Knowledge Graph
• Integrating Logical Rules Into Neural Multi-Hop Reasoning for Drug Repurposing
• Collaborative Policy Learning for Open Knowledge Graph Reasoning
• Knowledge Graph Embedding with Atrous Convolution and Residual Learning
• Semantic Graphs for Generating Deep Questions
• Search from History and Reason for Future: Two-stage Reasoning on Temporal Knowledge Graphs
• Programming and Deployment of Autonomous Swarms using Multi-Agent Reinforcement Learning
• Explainable Biomedical Recommendations via Reinforcement Learning Reasoning on Knowledge Graphs
• Learning to Order Sub-questions for Complex Question Answering
• Reasoning Over Paths via Knowledge Base Completion
• Combining Rules and Embeddings via Neuro-Symbolic AI for Knowledge Base Completion
• Recent Advances in Automated Question Answering In Biomedical Domain
• Navigation-Based Candidate Expansion and Pretrained Language Models for Citation Recommendation
• Knowledge Graph Completion with Text-aided Regularization