Mask and Reason: Pre-Training Knowledge Graph Transformers for Complex Logical Queries
arXiv:2208.07638 · doi:10.1145/3534678.3539472
Abstract
Knowledge graph (KG) embeddings have been a mainstream approach for reasoning over incomplete KGs. However, limited by their inherently shallow and static architectures, they can hardly deal with the rising focus on complex logical queries, which comprise logical operators, imputed edges, multiple source entities, and unknown intermediate entities. In this work, we present the Knowledge Graph Transformer (kgTransformer) with masked pre-training and fine-tuning strategies. We design a KG triple transformation method to enable Transformer to handle KGs, which is further strengthened by the Mixture-of-Experts (MoE) sparse activation. We then formulate the complex logical queries as masked prediction and introduce a two-stage masked pre-training strategy to improve transferability and generalizability. Extensive experiments on two benchmarks demonstrate that kgTransformer can consistently outperform both KG embedding-based baselines and advanced encoders on nine in-domain and out-of-domain reasoning tasks. Additionally, kgTransformer can reason with explainability via providing the full reasoning paths to interpret given answers.
arXiv commentkgTransformer; Accepted to KDD 2022
References in corpus (14)
- Decoupled Weight Decay Regularization
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
- Gaussian Error Linear Units (GELUs)
- Complex Embeddings for Simple Link Prediction
- Graph Contrastive Learning with Augmentations
- GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training
- Graph Transformer Networks
- Open Graph Benchmark: Datasets for Machine Learning on Graphs
- GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- Beta Embeddings for Multi-Hop Logical Reasoning in Knowledge Graphs
- GPT-GNN: Generative Pre-Training of Graph Neural Networks
- FastMoE: A Fast Mixture-of-Expert Training System
- Graph Random Neural Network for Semi-Supervised Learning on Graphs
- SMORE: Knowledge Graph Completion and Multi-hop Reasoning in Massive Knowledge Graphs