On the Global Self-attention Mechanism for Graph Convolutional Networks

Abstract

Applying Global Self-attention (GSA) mechanism over features has achieved remarkable success on Convolutional Neural Networks (CNNs). However, it is not clear if Graph Convolutional Networks (GCNs) can similarly benefit from such a technique. In this paper, inspired by the similarity between CNNs and GCNs, we study the impact of the Global Self-attention mechanism on GCNs. We find that consistent with the intuition, the GSA mechanism allows GCNs to capture feature-based vertex relations regardless of edge connections; As a result, the GSA mechanism can introduce extra expressive power to the GCNs. Furthermore, we analyze the impacts of the GSA mechanism on the issues of overfitting and over-smoothing. We prove that the GSA mechanism can alleviate both the overfitting and the over-smoothing issues based on some recent technical developments. Experiments on multiple benchmark datasets illustrate both superior expressive power and less significant overfitting and over-smoothing problems for the GSA-augmented GCNs, which corroborate the intuitions and the theoretical results.

References in corpus (15)

• Semi-Supervised Classification with Graph Convolutional Networks
• Inductive Representation Learning on Large Graphs
• Neural Message Passing for Quantum Chemistry
• Spectral Networks and Locally Connected Networks on Graphs
• Self-Attention Generative Adversarial Networks
• Deep Convolutional Networks on Graph-Structured Data
• Simplifying Graph Convolutional Networks
• Representation Learning on Graphs with Jumping Knowledge Networks
• DropEdge: Towards Deep Graph Convolutional Networks on Node Classification
• FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling
• Spatial-Temporal Transformer Networks for Traffic Flow Forecasting
• GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs
• Honeycombs and sums of Hermitian matrices
• Geom-GCN: Geometric Graph Convolutional Networks
• SAG-VAE: End-to-end Joint Inference of Data Representations and Feature Relations