Graph Neural Networks for Natural Language Processing: A Survey

Abstract

Deep learning has become the dominant approach in coping with various tasks in Natural LanguageProcessing (NLP). Although text inputs are typically represented as a sequence of tokens, there isa rich variety of NLP problems that can be best expressed with a graph structure. As a result, thereis a surge of interests in developing new deep learning techniques on graphs for a large numberof NLP tasks. In this survey, we present a comprehensive overview onGraph Neural Networks(GNNs) for Natural Language Processing. We propose a new taxonomy of GNNs for NLP, whichsystematically organizes existing research of GNNs for NLP along three axes: graph construction,graph representation learning, and graph based encoder-decoder models. We further introducea large number of NLP applications that are exploiting the power of GNNs and summarize thecorresponding benchmark datasets, evaluation metrics, and open-source codes. Finally, we discussvarious outstanding challenges for making the full use of GNNs for NLP as well as future researchdirections. To the best of our knowledge, this is the first comprehensive overview of Graph NeuralNetworks for Natural Language Processing.

127 pages, accepted by Foundations and Trends in Machine Learning

References in corpus (42)

• Sequence to Sequence Learning with Neural Networks
• Semi-Supervised Classification with Graph Convolutional Networks
• Distributed Representations of Sentences and Documents
• Language Models are Few-Shot Learners
• A Survey on Knowledge Graphs: Representation, Acquisition and Applications
• Deep Convolutional Networks on Graph-Structured Data
• Fast Graph Representation Learning with PyTorch Geometric
• Convolutional Neural Network Architectures for Matching Natural Language Sentences
• Linformer: Self-Attention with Linear Complexity
• Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
• Large-scale Simple Question Answering with Memory Networks
• Graph Transformer Networks
• Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention
• Iterative Deep Graph Learning for Graph Neural Networks: Better and Robust Node Embeddings
• Variational Reasoning for Question Answering with Knowledge Graph
• Random Feature Attention
• MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms
• Learning to Generate Questions by Learning What not to Generate
• Generating Wikipedia by Summarizing Long Sequences
• Retrieval-Augmented Generation for Code Summarization via Hybrid GNN
• A parallel corpus of Python functions and documentation strings for automated code documentation and code generation
• DIG: A Turnkey Library for Diving into Graph Deep Learning Research
• QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering
• Relational Graph Attention Network for Aspect-based Sentiment Analysis
• Multi-hop Question Generation with Graph Convolutional Network
• Representing Schema Structure with Graph Neural Networks for Text-to-SQL Parsing
• A General Framework for Information Extraction using Dynamic Span Graphs
• Every Document Owns Its Structure: Inductive Text Classification via Graph Neural Networks
• Visual Question Generation as Dual Task of Visual Question Answering
• Strongly Incremental Constituency Parsing with Graph Neural Networks
• Graph-to-Tree Neural Networks for Learning Structured Input-Output Translation with Applications to Semantic Parsing and Math Word Problem
• Leveraging Graph to Improve Abstractive Multi-Document Summarization
• Auto Completion of User Interface Layout Design Using Transformer-Based Tree Decoders
• AMR Parsing via Graph-Sequence Iterative Inference
• Structural Neural Encoders for AMR-to-text Generation
• Edge-Enhanced Graph Convolution Networks for Event Detection with Syntactic Relation
• Extracting Summary Knowledge Graphs from Long Documents
• Enhancing Extractive Text Summarization with Topic-Aware Graph Neural Networks
• Document Graph for Neural Machine Translation
• A Graph Reasoning Network for Multi-turn Response Selection via Customized Pre-training
• Coordinated Reasoning for Cross-Lingual Knowledge Graph Alignment
• Stronger Transformers for Neural Multi-Hop Question Generation

Cited by in corpus (4)

• Deep reinforcement learning for machine scheduling: Methodology, the state-of-the-art, and future directions
• Evaluating Deep Graph Neural Networks
• GNNBuilder: An Automated Framework for Generic Graph Neural Network Accelerator Generation, Simulation, and Optimization
• Graph-augmented Learning to Rank for Querying Large-scale Knowledge Graph