Attention-based graph neural networks: a survey

Abstract

Graph neural networks (GNNs) aim to learn well-trained representations in a lower-dimension space for downstream tasks while preserving the topological structures. In recent years, attention mechanism, which is brilliant in the fields of natural language processing and computer vision, is introduced to GNNs to adaptively select the discriminative features and automatically filter the noisy information. To the best of our knowledge, due to the fast-paced advances in this domain, a systematic overview of attention-based GNNs is still missing. To fill this gap, this paper aims to provide a comprehensive survey on recent advances in attention-based GNNs. Firstly, we propose a novel two-level taxonomy for attention-based GNNs from the perspective of development history and architectural perspectives. Specifically, the upper level reveals the three developmental stages of attention-based GNNs, including graph recurrent attention networks, graph attention networks, and graph transformers. The lower level focuses on various typical architectures of each stage. Secondly, we review these attention-based methods following the proposed taxonomy in detail and summarize the advantages and disadvantages of various models. A model characteristics table is also provided for a more comprehensive comparison. Thirdly, we share our thoughts on some open issues and future directions of attention-based GNNs. We hope this survey will provide researchers with an up-to-date reference regarding applications of attention-based GNNs. In addition, to cope with the rapid development in this field, we intend to share the relevant latest papers as an open resource at https://github.com/sunxiaobei/awesome-attention-based-gnns.

This is the accepted manuscript of an article published in Artificial Intelligence Review. The final version is available online at: [10.1007/s10462-023-10577-2](https://link.springer.com/article/10.1007/s10462-023-10577-2)

References in corpus (26)

• A Comprehensive Survey on Graph Neural Networks
• DeepWalk: Online Learning of Social Representations
• LINE: Large-scale Information Network Embedding
• KGAT: Knowledge Graph Attention Network for Recommendation
• Graph Neural Network for Traffic Forecasting: A Survey
• The physics of higher-order interactions in complex systems
• Hierarchical Graph Representation Learning with Differentiable Pooling
• A General Survey on Attention Mechanisms in Deep Learning
• Towards Deeper Graph Neural Networks
• AM-GCN: Adaptive Multi-channel Graph Convolutional Networks
• Representation Learning for Attributed Multiplex Heterogeneous Network
• Graph Learning: A Survey
• Predict then Propagate: Graph Neural Networks meet Personalized PageRank
• A Comprehensive Survey on Community Detection with Deep Learning
• A Generalization of Transformer Networks to Graphs
• Dual Graph Attention Networks for Deep Latent Representation of Multifaceted Social Effects in Recommender Systems
• Efficient Transformers: A Survey
• Combinatorial Optimization with Physics-Inspired Graph Neural Networks
• Gated Graph Recurrent Neural Networks
• Double-Scale Self-Supervised Hypergraph Learning for Group Recommendation
• Graph Attention Multi-Layer Perceptron
• DisenKGAT: Knowledge Graph Embedding with Disentangled Graph Attention Network
• Breaking the Limit of Graph Neural Networks by Improving the Assortativity of Graphs with Local Mixing Patterns
• Improving Attention Mechanism in Graph Neural Networks via Cardinality Preservation
• Graph Convolutional Network-based Feature Selection for High-dimensional and Low-sample Size Data
• Path-Augmented Graph Transformer Network

Cited by in corpus (1)

• PolyFormer: Scalable Node-wise Filters via Polynomial Graph Transformer