Self-Supervised Graph Transformer on Large-Scale Molecular Data

Abstract

How to obtain informative representations of molecules is a crucial prerequisite in AI-driven drug design and discovery. Recent researches abstract molecules as graphs and employ Graph Neural Networks (GNNs) for molecular representation learning. Nevertheless, two issues impede the usage of GNNs in real scenarios: (1) insufficient labeled molecules for supervised training; (2) poor generalization capability to new-synthesized molecules. To address them both, we propose a novel framework, GROVER, which stands for Graph Representation frOm self-superVised mEssage passing tRansformer. With carefully designed self-supervised tasks in node-, edge- and graph-level, GROVER can learn rich structural and semantic information of molecules from enormous unlabelled molecular data. Rather, to encode such complex information, GROVER integrates Message Passing Networks into the Transformer-style architecture to deliver a class of more expressive encoders of molecules. The flexibility of GROVER allows it to be trained efficiently on large-scale molecular dataset without requiring any supervision, thus being immunized to the two issues mentioned above. We pre-train GROVER with 100 million parameters on 10 million unlabelled molecules -- the biggest GNN and the largest training dataset in molecular representation learning. We then leverage the pre-trained GROVER for molecular property prediction followed by task-specific fine-tuning, where we observe a huge improvement (more than 6% on average) from current state-of-the-art methods on 11 challenging benchmarks. The insights we gained are that well-designed self-supervision losses and largely-expressive pre-trained models enjoy the significant potential on performance boosting.

arXiv comment17 pages, 7 figures

References in corpus (15)

• Sequence to Sequence Learning with Neural Networks
• Inductive Representation Learning on Large Graphs
• Neural Message Passing for Quantum Chemistry
• Molecular Graph Convolutions: Moving Beyond Fingerprints
• Quantum-Chemical Insights from Deep Tensor Neural Networks
• DropEdge: Towards Deep Graph Convolutional Networks on Node Classification
• AtomNet: A Deep Convolutional Neural Network for Bioactivity Prediction in Structure-based Drug Discovery
• Horovod: fast and easy distributed deep learning in TensorFlow
• Massively Multitask Networks for Drug Discovery
• InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization
• N-Gram Graph: Simple Unsupervised Representation for Graphs, with Applications to Molecules
• Deeply learning molecular structure-property relationships using attention- and gate-augmented graph convolutional network
• Tackling Over-Smoothing for General Graph Convolutional Networks
• Learning to SMILE(S)
• Rumor Detection on Social Media with Bi-Directional Graph Convolutional Networks

Cited by in corpus (23)

• ChemRL-GEM: Geometry Enhanced Molecular Representation Learning for Property Prediction
• Graph Self-Supervised Learning: A Survey
• SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation
• Global Self-Attention as a Replacement for Graph Convolution
• A Comprehensive Survey on Trustworthy Graph Neural Networks: Privacy, Robustness, Fairness, and Explainability
• Improving Molecular Contrastive Learning via Faulty Negative Mitigation and Decomposed Fragment Contrast
• KPGT: Knowledge-Guided Pre-training of Graph Transformer for Molecular Property Prediction
• Bidirectional Generation of Structure and Properties Through a Single Molecular Foundation Model
• Roadmap on machine learning glassy dynamics
• Drug-Target Interaction/Affinity Prediction: Deep Learning Models and Advances Review
• Graph Neural Networks: Methods, Applications, and Opportunities
• Automated 3D Pre-Training for Molecular Property Prediction
• Adversarial Attack Framework on Graph Embedding Models with Limited Knowledge
• Multi-channel learning for integrating structural hierarchies into context-dependent molecular representation
• Motif-Driven Contrastive Learning of Graph Representations
• A Large Encoder-Decoder Family of Foundation Models For Chemical Language
• BatmanNet: Bi-branch Masked Graph Transformer Autoencoder for Molecular Representation
• Multiresolution Graph Transformers and Wavelet Positional Encoding for Learning Hierarchical Structures
• Artificial Intelligence in Drug Discovery: Applications and Techniques
• PEMP: Leveraging Physics Properties to Enhance Molecular Property Prediction
• Heterogeneity-aware Cross-school Electives Recommendation: a Hybrid Federated Approach
• Dual-Modality Representation Learning for Molecular Property Prediction
• Hybrid Focal and Full-Range Attention Based Graph Transformers