Graph-Bert: Only Attention is Needed for Learning Graph Representations

Abstract

The dominant graph neural networks (GNNs) over-rely on the graph links, several serious performance problems with which have been witnessed already, e.g., suspended animation problem and over-smoothing problem. What's more, the inherently inter-connected nature precludes parallelization within the graph, which becomes critical for large-sized graph, as memory constraints limit batching across the nodes. In this paper, we will introduce a new graph neural network, namely GRAPH-BERT (Graph based BERT), solely based on the attention mechanism without any graph convolution or aggregation operators. Instead of feeding GRAPH-BERT with the complete large input graph, we propose to train GRAPH-BERT with sampled linkless subgraphs within their local contexts. GRAPH-BERT can be learned effectively in a standalone mode. Meanwhile, a pre-trained GRAPH-BERT can also be transferred to other application tasks directly or with necessary fine-tuning if any supervised label information or certain application oriented objective is available. We have tested the effectiveness of GRAPH-BERT on several graph benchmark datasets. Based the pre-trained GRAPH-BERT with the node attribute reconstruction and structure recovery tasks, we further fine-tune GRAPH-BERT on node classification and graph clustering tasks specifically. The experimental results have demonstrated that GRAPH-BERT can out-perform the existing GNNs in both the learning effectiveness and efficiency.

10 pages

References in corpus (7)

• Inductive Representation Learning on Large Graphs
• Junction Tree Variational Autoencoder for Molecular Graph Generation
• Learning Convolutional Neural Networks for Graphs
• The Anatomy of the Facebook Social Graph
• GResNet: Graph Residual Network for Reviving Deep GNNs from Suspended Animation
• IsoNN: Isomorphic Neural Network for Graph Representation Learning and Classification
• Deep Loopy Neural Network Model for Graph Structured Data Representation Learning

Cited by in corpus (36)

• Graph Contrastive Learning with Augmentations
• Graph Self-Supervised Learning: A Survey
• A Comprehensive Survey on Deep Graph Representation Learning
• A Generalization of Transformer Networks to Graphs
• Graph Neural Networks: Taxonomy, Advances and Trends
• Do Transformers Really Perform Bad for Graph Representation?
• Global Self-Attention as a Replacement for Graph Convolution
• Self-Supervised Hyperboloid Representations from Logical Queries over Knowledge Graphs
• GraphiT: Encoding Graph Structure in Transformers
• Self-supervised Auxiliary Learning with Meta-paths for Heterogeneous Graphs
• Multi-hop Attention Graph Neural Network
• Graph Neural Networks: Methods, Applications, and Opportunities
• Sub-graph Contrast for Scalable Self-Supervised Graph Representation Learning
• SLAPS: Self-Supervision Improves Structure Learning for Graph Neural Networks
• Logiformer: A Two-Branch Graph Transformer Network for Interpretable Logical Reasoning
• Improving Pre-trained Language Model Fine-tuning with Noise Stability Regularization
• Sampling methods for efficient training of graph convolutional networks: A survey
• VQ-GNN: A Universal Framework to Scale up Graph Neural Networks using Vector Quantization
• NodeNet: A Graph Regularised Neural Network for Node Classification
• Initialization Matters: Regularizing Manifold-informed Initialization for Neural Recommendation Systems
• Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction
• Few-Shot Electronic Health Record Coding through Graph Contrastive Learning
• Segmented Graph-Bert for Graph Instance Modeling
• Pre-training Graph Transformer with Multimodal Side Information for Recommendation
• DyFormer: A Scalable Dynamic Graph Transformer with Provable Benefits on Generalization Ability
• Graph Neural Distance Metric Learning with Graph-Bert
• LUKE-Graph: A Transformer-based Approach with Gated Relational Graph Attention for Cloze-style Reading Comprehension
• TransCamP: Graph Transformer for 6-DoF Camera Pose Estimation
• G5: A Universal GRAPH-BERT for Graph-to-Graph Transfer and Apocalypse Learning
• Predicting times of waiting on red signals using BERT
• Hybrid Focal and Full-Range Attention Based Graph Transformers
• Multimodal Graph-based Transformer Framework for Biomedical Relation Extraction
• On Inductive Biases for Machine Learning in Data Constrained Settings
• Multi-hop Graph Convolutional Network with High-order Chebyshev Approximation for Text Reasoning
• Network representation learning: A macro and micro view
• Structured Context and High-Coverage Grammar for Conversational Question Answering over Knowledge Graphs