Neural Graph Embedding Methods for Natural Language Processing

Abstract

Knowledge graphs are structured representations of facts in a graph, where nodes represent entities and edges represent relationships between them. Recent research has resulted in the development of several large KGs. However, all of them tend to be sparse with very few facts per entity. In the first part of the thesis, we propose two solutions to alleviate this problem: (1) KG Canonicalization, i.e., identifying and merging duplicate entities in a KG, (2) Relation Extraction which involves automating the process of extracting semantic relationships between entities from unstructured text. Traditional Neural Networks like CNNs and RNNs are constrained to handle Euclidean data. However, graphs in Natural Language Processing (NLP) are prominent. Recently, Graph Convolutional Networks (GCNs) have been proposed to address this shortcoming and have been successfully applied for several problems. In the second part of the thesis, we utilize GCNs for Document Timestamping problem and for learning word embeddings using dependency context of a word instead of sequential context. In this third part of the thesis, we address two limitations of existing GCN models, i.e., (1) The standard neighborhood aggregation scheme puts no constraints on the number of nodes that can influence the representation of a target node. This leads to a noisy representation of hub-nodes which coves almost the entire graph in a few hops. (2) Most of the existing GCN models are limited to handle undirected graphs. However, a more general and pervasive class of graphs are relational graphs where each edge has a label and direction associated with it. Existing approaches to handle such graphs suffer from over-parameterization and are restricted to learning representation of nodes only.

arXiv comment168 pages, PhD thesis (2019)

References in corpus (18)

• Adam: A Method for Stochastic Optimization
• Distributed Representations of Words and Phrases and their Compositionality
• Sequence to Sequence Learning with Neural Networks
• Natural Language Processing (almost) from Scratch
• The Emerging Field of Signal Processing on Graphs: Extending High-Dimensional Data Analysis to Networks and Other Irregular Domains
• Neural Message Passing for Quantum Chemistry
• Geometric deep learning: going beyond Euclidean data
• A Review of Relational Machine Learning for Knowledge Graphs
• Representation Learning on Graphs with Jumping Knowledge Networks
• RotatE: Knowledge Graph Embedding by Relational Rotation in Complex Space
• Learning Deep Generative Models of Graphs
• Hypernetwork Knowledge Graph Embeddings
• Word Representations via Gaussian Embedding
• Probabilistic Similarity Logic
• Improving Distantly Supervised Relation Extraction using Word and Entity Based Attention
• Graph Partition Neural Networks for Semi-Supervised Classification
• Confidence-based Graph Convolutional Networks for Semi-Supervised Learning
• SEE: Syntax-aware Entity Embedding for Neural Relation Extraction

Cited by in corpus (1)

• Analyzing the Performance of Graph Neural Networks with Pipe Parallelism