Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction

Abstract

Learning on graphs has attracted significant attention in the learning community due to numerous real-world applications. In particular, graph neural networks (GNNs), which take numerical node features and graph structure as inputs, have been shown to achieve state-of-the-art performance on various graph-related learning tasks. Recent works exploring the correlation between numerical node features and graph structure via self-supervised learning have paved the way for further performance improvements of GNNs. However, methods used for extracting numerical node features from raw data are still graph-agnostic within standard GNN pipelines. This practice is sub-optimal as it prevents one from fully utilizing potential correlations between graph topology and node attributes. To mitigate this issue, we propose a new self-supervised learning framework, Graph Information Aided Node feature exTraction (GIANT). GIANT makes use of the eXtreme Multi-label Classification (XMC) formalism, which is crucial for fine-tuning the language model based on graph information, and scales to large datasets. We also provide a theoretical analysis that justifies the use of XMC over link prediction and motivates integrating XR-Transformers, a powerful method for solving XMC problems, into the GIANT framework. We demonstrate the superior performance of GIANT over the standard GNN pipeline on Open Graph Benchmark datasets: For example, we improve the accuracy of the top-ranked method GAMLP from $68.25\%$ to $69.67\%$, SGC from $63.29\%$ to $66.10\%$ and MLP from $47.24\%$ to $61.10\%$ on the ogbn-papers100M dataset by leveraging GIANT.

Published in ICLR 2022

References in corpus (25)

• Distributed Representations of Words and Phrases and their Compositionality
• Link Prediction in Complex Networks: A Survey
• Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks
• Fast Graph Representation Learning with PyTorch Geometric
• Simplifying Graph Convolutional Networks
• Variational Graph Auto-Encoders
• GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Models
• Strategies for Pre-training Graph Neural Networks
• Graph-Bert: Only Attention is Needed for Learning Graph Representations
• Geom-GCN: Geometric Graph Convolutional Networks
• Combining Label Propagation and Simple Models Out-performs Graph Neural Networks
• Graph Attention Multi-Layer Perceptron
• Pre-training Tasks for Embedding-based Large-scale Retrieval
• Self-Supervised Generalisation with Meta Auxiliary Learning
• AttentionXML: Label Tree-based Attention-Aware Deep Model for High-Performance Extreme Multi-Label Text Classification
• Contextual Stochastic Block Models
• PECOS: Prediction for Enormous and Correlated Output Spaces
• New Benchmarks for Learning on Non-Homophilous Graphs
• Optimizing Generalized PageRank Methods for Seed-Expansion Community Detection
• Scalable and Adaptive Graph Neural Networks with Self-Label-Enhanced training
• LightXML: Transformer with Dynamic Negative Sampling for High-Performance Extreme Multi-label Text Classification
• Graph Convolution for Semi-Supervised Classification: Improved Linear Separability and Out-of-Distribution Generalization
• Training Graph Neural Networks with 1000 Layers
• Label Disentanglement in Partition-based Extreme Multilabel Classification
• Extreme Multi-label Classification from Aggregated Labels