JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphs

Abstract

Existing pre-trained models for knowledge-graph-to-text (KG-to-text) generation simply fine-tune text-to-text pre-trained models such as BART or T5 on KG-to-text datasets, which largely ignore the graph structure during encoding and lack elaborate pre-training tasks to explicitly model graph-text alignments. To tackle these problems, we propose a graph-text joint representation learning model called JointGT. During encoding, we devise a structure-aware semantic aggregation module which is plugged into each Transformer layer to preserve the graph structure. Furthermore, we propose three new pre-training tasks to explicitly enhance the graph-text alignment including respective text / graph reconstruction, and graph-text alignment in the embedding space via Optimal Transport. Experiments show that JointGT obtains new state-of-the-art performance on various KG-to-text datasets.

arXiv commentACL 2021 (Findings)

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Distributed Representations of Words and Phrases and their Compositionality
• An Interpretable Reasoning Network for Multi-Relation Question Answering
• Toward Subgraph-Guided Knowledge Graph Question Generation with Graph Neural Networks
• A Fast Proximal Point Method for Computing Exact Wasserstein Distance
• Modeling Global and Local Node Contexts for Text Generation from Knowledge Graphs
• CycleGT: Unsupervised Graph-to-Text and Text-to-Graph Generation via Cycle Training
• Text-to-Text Pre-Training for Data-to-Text Tasks
• JAKET: Joint Pre-training of Knowledge Graph and Language Understanding
• Graph Optimal Transport for Cross-Domain Alignment