Molecular Contrastive Learning of Representations via Graph Neural Networks

Abstract

Molecular Machine Learning (ML) bears promise for efficient molecule property prediction and drug discovery. However, labeled molecule data can be expensive and time-consuming to acquire. Due to the limited labeled data, it is a great challenge for supervised-learning ML models to generalize to the giant chemical space. In this work, we present MolCLR: Molecular Contrastive Learning of Representations via Graph Neural Networks (GNNs), a self-supervised learning framework that leverages large unlabeled data (~10M unique molecules). In MolCLR pre-training, we build molecule graphs and develop GNN encoders to learn differentiable representations. Three molecule graph augmentations are proposed: atom masking, bond deletion, and subgraph removal. A contrastive estimator maximizes the agreement of augmentations from the same molecule while minimizing the agreement of different molecules. Experiments show that our contrastive learning framework significantly improves the performance of GNNs on various molecular property benchmarks including both classification and regression tasks. Benefiting from pre-training on the large unlabeled database, MolCLR even achieves state-of-the-art on several challenging benchmarks after fine-tuning. Additionally, further investigations demonstrate that MolCLR learns to embed molecules into representations that can distinguish chemically reasonable molecular similarities.

References in corpus (24)

• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• Neural Message Passing for Quantum Chemistry
• Geometric deep learning: going beyond Euclidean data
• Automatic chemical design using a data-driven continuous representation of molecules
• On representing chemical environments
• SchNet - a deep learning architecture for molecules and materials
• Molecular Transformer - A Model for Uncertainty-Calibrated Chemical Reaction Prediction
• Convolutional Networks on Graphs for Learning Molecular Fingerprints
• Graph Contrastive Learning with Augmentations
• GuacaMol: Benchmarking Models for De Novo Molecular Design
• Towards Exact Molecular Dynamics Simulations with Machine-Learned Force Fields
• Self-Referencing Embedded Strings (SELFIES): A 100% robust molecular string representation
• Graph Transformer Networks
• ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction
• Massively Multitask Networks for Drug Discovery
• Directional Message Passing for Molecular Graphs
• Understanding molecular representations in machine learning: The role of uniqueness and target similarity
• Analyzing Learned Molecular Representations for Property Prediction
• Coarse-Graining Auto-Encoders for Molecular Dynamics
• Orbital Graph Convolutional Neural Network for Material Property Prediction
• Hierarchical Generation of Molecular Graphs using Structural Motifs
• Molecule Attention Transformer
• Deep Reinforcement Learning Optimizes Graphene Nanopores for Efficient Desalination

Cited by in corpus (37)

• A Comprehensive Survey on Deep Graph Representation Learning
• polyBERT: A chemical language model to enable fully machine-driven ultrafast polymer informatics
• TransPolymer: a Transformer-based language model for polymer property predictions
• MechRetro is a chemical-mechanism-driven graph learning framework for interpretable retrosynthesis prediction and pathway planning
• Comprehensive evaluation of deep and graph learning on drug-drug interactions prediction
• Improving Molecular Contrastive Learning via Faulty Negative Mitigation and Decomposed Fragment Contrast
• Graph Neural Networks for Molecules
• GIT-Mol: A Multi-modal Large Language Model for Molecular Science with Graph, Image, and Text
• End-to-end Wind Turbine Wake Modelling with Deep Graph Representation Learning
• Artificial Intelligence in Material Engineering: A review on applications of AI in Material Engineering
• Graph Neural Networks Accelerated Molecular Dynamics
• Denoise Pretraining on Nonequilibrium Molecules for Accurate and Transferable Neural Potentials
• Geometry-enhanced Pre-training on Interatomic Potentials
• Multi-channel learning for integrating structural hierarchies into context-dependent molecular representation
• Surrogate- and invariance-boosted contrastive learning for data-scarce applications in science
• MolSets: Molecular Graph Deep Sets Learning for Mixture Property Modeling
• Spiers Memorial Lecture: How to do impactful research in artificial intelligence for chemistry and materials science
• Discovering dynamic laws from observations: the case of self-propelled, interacting colloids
• Artificial Intelligence for Science in Quantum, Atomistic, and Continuum Systems
• A Large Encoder-Decoder Family of Foundation Models For Chemical Language
• Counterfactual Learning on Graphs: A Survey
• Benchmarking Large Language Models for Polymer Property Predictions
• Beyond Independent Error Assumptions in Large GNN Atomistic Models
• Beyond Over-smoothing: Uncovering the Trainability Challenges in Deep Graph Neural Networks
• Contrastive Dual-Interaction Graph Neural Network for Molecular Property Prediction
• HD-Bind: Encoding of Molecular Structure with Low Precision, Hyperdimensional Binary Representations
• Chemical Language Model Linker: blending text and molecules with modular adapters
• A Survey on Learning from Graphs with Heterophily: Recent Advances and Future Directions
• A simple framework for contrastive learning phases of matter
• Handling Feature Heterogeneity with Learnable Graph Patches
• Multi-Level Fusion Graph Neural Network for Molecule Property Prediction
• Pretraining a Foundation Model for Small-Molecule Natural Products
• A Machine Learning Benchmarking Framework for Lipid Nanoparticle Transfection Efficiency Prediction
• Introduction to Graph Neural Networks for Machine Learning Engineers
• PASCL: Supervised Contrastive Learning with Perturbative Augmentation for Particle Decay Reconstruction
• Dual-Modality Representation Learning for Molecular Property Prediction
• M-JEPA: Predictive Self-Supervised Learning for Molecular Graphs with Scaffold-Shift Evaluation on Tox21