Global Self-Attention as a Replacement for Graph Convolution

Abstract

We propose an extension to the transformer neural network architecture for general-purpose graph learning by adding a dedicated pathway for pairwise structural information, called edge channels. The resultant framework - which we call Edge-augmented Graph Transformer (EGT) - can directly accept, process and output structural information of arbitrary form, which is important for effective learning on graph-structured data. Our model exclusively uses global self-attention as an aggregation mechanism rather than static localized convolutional aggregation. This allows for unconstrained long-range dynamic interactions between nodes. Moreover, the edge channels allow the structural information to evolve from layer to layer, and prediction tasks on edges/links can be performed directly from the output embeddings of these channels. We verify the performance of EGT in a wide range of graph-learning experiments on benchmark datasets, in which it outperforms Convolutional/Message-Passing Graph Neural Networks. EGT sets a new state-of-the-art for the quantum-chemical regression task on the OGB-LSC PCQM4Mv2 dataset containing 3.8 million molecular graphs. Our findings indicate that global self-attention based aggregation can serve as a flexible, adaptive and effective replacement of graph convolution for general-purpose graph learning. Therefore, convolutional local neighborhood aggregation is not an essential inductive bias.

The accepted version in KDD '22

References in corpus (7)

• Graph Transformer Networks
• A Generalization of Transformer Networks to Graphs
• DeeperGCN: All You Need to Train Deeper GCNs
• Graph-Bert: Only Attention is Needed for Learning Graph Representations
• Do Transformers Really Perform Bad for Graph Representation?
• OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs
• Graph Transformer for Graph-to-Sequence Learning

Cited by in corpus (6)

• Automated 3D Pre-Training for Molecular Property Prediction
• Graph Attention-Based Symmetry Constraint Extraction for Analog Circuits
• Masked Graph Transformer for Large-Scale Recommendation
• CHILI: Chemically-Informed Large-scale Inorganic Nanomaterials Dataset for Advancing Graph Machine Learning
• The Information Pathways Hypothesis: Transformers are Dynamic Self-Ensembles
• Hybrid Focal and Full-Range Attention Based Graph Transformers