Graph Neural Networks for Natural Language Processing: A Survey

Abstract

Deep learning has become the dominant approach in coping with various tasks in Natural LanguageProcessing (NLP). Although text inputs are typically represented as a sequence of tokens, there isa rich variety of NLP problems that can be best expressed with a graph structure. As a result, thereis a surge of interests in developing new deep learning techniques on graphs for a large numberof NLP tasks. In this survey, we present a comprehensive overview onGraph Neural Networks(GNNs) for Natural Language Processing. We propose a new taxonomy of GNNs for NLP, whichsystematically organizes existing research of GNNs for NLP along three axes: graph construction,graph representation learning, and graph based encoder-decoder models. We further introducea large number of NLP applications that are exploiting the power of GNNs and summarize thecorresponding benchmark datasets, evaluation metrics, and open-source codes. Finally, we discussvarious outstanding challenges for making the full use of GNNs for NLP as well as future researchdirections. To the best of our knowledge, this is the first comprehensive overview of Graph NeuralNetworks for Natural Language Processing.

arXiv comment127 pages, accepted by Foundations and Trends in Machine Learning

References in corpus (81)

• Distributed Representations of Words and Phrases and their Compositionality
• Sequence to Sequence Learning with Neural Networks
• Semi-Supervised Classification with Graph Convolutional Networks
• YOLOv3: An Incremental Improvement
• Distributed Representations of Sentences and Documents
• Inductive Representation Learning on Large Graphs
• Neural Message Passing for Quantum Chemistry
• Character-level Convolutional Networks for Text Classification
• Language Models are Few-Shot Learners
• A Survey on Knowledge Graphs: Representation, Acquisition and Applications
• Teaching Machines to Read and Comprehend
• Convolutional Sequence to Sequence Learning
• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Bidirectional Attention Flow for Machine Comprehension
• Deep Convolutional Networks on Graph-Structured Data
• Fast Graph Representation Learning with PyTorch Geometric
• A Deep Reinforced Model for Abstractive Summarization
• Recurrent Neural Network for Text Classification with Multi-Task Learning
• Convolutional Neural Network Architectures for Matching Natural Language Sentences
• Linformer: Self-Attention with Linear Complexity
• Hierarchical Graph Representation Learning with Differentiable Pooling
• ConceptNet 5.5: An Open Multilingual Graph of General Knowledge
• Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
• End-To-End Memory Networks
• Large-scale Simple Question Answering with Memory Networks
• Graph Transformer Networks
• Dynamic Coattention Networks For Question Answering
• A large annotated corpus for learning natural language inference
• Gated Graph Sequence Neural Networks
• Jointly Multiple Events Extraction via Attention-based Graph Information Aggregation
• Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
• Neural Natural Language Inference Models Enhanced with External Knowledge
• Iterative Deep Graph Learning for Graph Neural Networks: Better and Robust Node Embeddings
• ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
• Learning Discrete Structures for Graph Neural Networks
• Variational Reasoning for Question Answering with Knowledge Graph
• A Deep Architecture for Semantic Matching with Multiple Positional Sentence Representations
• Structured Neural Summarization
• Graph2Seq: Graph to Sequence Learning with Attention-based Neural Networks
• Modeling Coverage for Neural Machine Translation
• Random Feature Attention
• MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms
• Language to Logical Form with Neural Attention
• Learning to Generate Questions by Learning What not to Generate
• Abstract Meaning Representation for Multi-Document Summarization
• Reading Wikipedia to Answer Open-Domain Questions
• Exploring Graph-structured Passage Representation for Multi-hop Reading Comprehension with Graph Neural Networks
• Generating Wikipedia by Summarizing Long Sequences
• Learning to Represent Edits
• Retrieval-Augmented Generation for Code Summarization via Hybrid GNN
• GraphIE: A Graph-Based Framework for Information Extraction
• A parallel corpus of Python functions and documentation strings for automated code documentation and code generation
• DIG: A Turnkey Library for Diving into Graph Deep Learning Research
• Modeling Semantics with Gated Graph Neural Networks for Knowledge Base Question Answering
• Large Scale Graph Learning from Smooth Signals
• QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering
• Relational Graph Attention Network for Aspect-based Sentiment Analysis
• Reinforcement Learning Based Graph-to-Sequence Model for Natural Question Generation
• Multi-hop Question Generation with Graph Convolutional Network
• Representing Schema Structure with Graph Neural Networks for Text-to-SQL Parsing
• A General Framework for Information Extraction using Dynamic Span Graphs
• A Benchmarking Environment for Reinforcement Learning Based Task Oriented Dialogue Management
• Every Document Owns Its Structure: Inductive Text Classification via Graph Neural Networks
• Visual Question Generation as Dual Task of Visual Question Answering
• Strongly Incremental Constituency Parsing with Graph Neural Networks
• Neural Document Summarization by Jointly Learning to Score and Select Sentences
• Jointly Predicting Predicates and Arguments in Neural Semantic Role Labeling
• Table-to-Text: Describing Table Region with Natural Language
• Graph-to-Tree Neural Networks for Learning Structured Input-Output Translation with Applications to Semantic Parsing and Math Word Problem
• Dynamically Pruned Message Passing Networks for Large-Scale Knowledge Graph Reasoning
• Leveraging Graph to Improve Abstractive Multi-Document Summarization
• Auto Completion of User Interface Layout Design Using Transformer-Based Tree Decoders
• Structural Neural Encoders for AMR-to-text Generation
• AMR Parsing via Graph-Sequence Iterative Inference
• Edge-Enhanced Graph Convolution Networks for Event Detection with Syntactic Relation
• Extracting Summary Knowledge Graphs from Long Documents
• Enhancing Extractive Text Summarization with Topic-Aware Graph Neural Networks
• Document Graph for Neural Machine Translation
• A Graph Reasoning Network for Multi-turn Response Selection via Customized Pre-training
• Stronger Transformers for Neural Multi-Hop Question Generation
• Coordinated Reasoning for Cross-Lingual Knowledge Graph Alignment

Cited by in corpus (4)

• Deep reinforcement learning for machine scheduling: Methodology, the state-of-the-art, and future directions
• Evaluating Deep Graph Neural Networks
• GNNBuilder: An Automated Framework for Generic Graph Neural Network Accelerator Generation, Simulation, and Optimization
• Graph-augmented Learning to Rank for Querying Large-scale Knowledge Graph