Multi-hop Attention Graph Neural Network

Abstract

Self-attention mechanism in graph neural networks (GNNs) led to state-of-the-art performance on many graph representation learning tasks. Currently, at every layer, attention is computed between connected pairs of nodes and depends solely on the representation of the two nodes. However, such attention mechanism does not account for nodes that are not directly connected but provide important network context. Here we propose Multi-hop Attention Graph Neural Network (MAGNA), a principled way to incorporate multi-hop context information into every layer of attention computation. MAGNA diffuses the attention scores across the network, which increases the receptive field for every layer of the GNN. Unlike previous approaches, MAGNA uses a diffusion prior on attention values, to efficiently account for all paths between the pair of disconnected nodes. We demonstrate in theory and experiments that MAGNA captures large-scale structural information in every layer, and has a low-pass effect that eliminates noisy high-frequency information from graph data. Experimental results on node classification as well as the knowledge graph completion benchmarks show that MAGNA achieves state-of-the-art results: MAGNA achieves up to 5.7 percent relative error reduction over the previous state-of-the-art on Cora, Citeseer, and Pubmed. MAGNA also obtains the best performance on a large-scale Open Graph Benchmark dataset. On knowledge graph completion MAGNA advances state-of-the-art on WN18RR and FB15k-237 across four different performance metrics.

arXiv commentAccepted by IJCAI 2021

References in corpus (12)

• Discrete Signal Processing on Graphs
• Representation Learning on Graphs with Jumping Knowledge Networks
• Open Graph Benchmark: Datasets for Machine Learning on Graphs
• Quaternion Knowledge Graph Embeddings
• GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphs
• Attention-based Graph Neural Network for Semi-supervised Learning
• LanczosNet: Multi-Scale Deep Graph Convolutional Networks
• Graph-Bert: Only Attention is Needed for Learning Graph Representations
• CoKE: Contextualized Knowledge Graph Embedding
• GeniePath: Graph Neural Networks with Adaptive Receptive Paths
• Improving Graph Attention Networks with Large Margin-based Constraints
• Tree Transformer: Integrating Tree Structures into Self-Attention

Cited by in corpus (6)

• A Comprehensive Survey on Deep Graph Representation Learning
• Do Transformers Really Perform Bad for Graph Representation?
• Should Graph Convolution Trust Neighbors? A Simple Causal Inference Method
• Graph Ensemble Learning over Multiple Dependency Trees for Aspect-level Sentiment Classification
• Tree Decomposed Graph Neural Network
• Spectral Graph Attention Network with Fast Eigen-approximation