KPGT: Knowledge-Guided Pre-training of Graph Transformer for Molecular Property Prediction
arXiv:2206.03364 · doi:10.1145/3534678.3539426
Abstract
Designing accurate deep learning models for molecular property prediction plays an increasingly essential role in drug and material discovery. Recently, due to the scarcity of labeled molecules, self-supervised learning methods for learning generalizable and transferable representations of molecular graphs have attracted lots of attention. In this paper, we argue that there exist two major issues hindering current self-supervised learning methods from obtaining desired performance on molecular property prediction, that is, the ill-defined pre-training tasks and the limited model capacity. To this end, we introduce Knowledge-guided Pre-training of Graph Transformer (KPGT), a novel self-supervised learning framework for molecular graph representation learning, to alleviate the aforementioned issues and improve the performance on the downstream molecular property prediction tasks. More specifically, we first introduce a high-capacity model, named Line Graph Transformer (LiGhT), which emphasizes the importance of chemical bonds and is mainly designed to model the structural information of molecular graphs. Then, a knowledge-guided pre-training strategy is proposed to exploit the additional knowledge of molecules to guide the model to capture the abundant structural and semantic information from large-scale unlabeled molecular graphs. Extensive computational tests demonstrated that KPGT can offer superior performance over current state-of-the-art methods on several molecular property prediction tasks.
arXiv comment11 pages, to appear in KDD 2022 research track
References in corpus (16)
- Adam: A Method for Stochastic Optimization
- Overcoming catastrophic forgetting in neural networks
- Neural Message Passing for Quantum Chemistry
- Gaussian Error Linear Units (GELUs)
- Deep Learning using Rectified Linear Units (ReLU)
- Graph Contrastive Learning with Augmentations
- Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks
- Beyond English-Centric Multilingual Machine Translation
- Self-Supervised Graph Transformer on Large-Scale Molecular Data
- Principal Neighbourhood Aggregation for Graph Nets
- Analyzing Learned Molecular Representations for Property Prediction
- Pre-training Molecular Graph Representation with 3D Geometry
- Molecule Attention Transformer
- Graph Contrastive Learning Automated
- GraphiT: Encoding Graph Structure in Transformers
- Path-Augmented Graph Transformer Network