ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction

Abstract

GNNs and chemical fingerprints are the predominant approaches to representing molecules for property prediction. However, in NLP, transformers have become the de-facto standard for representation learning thanks to their strong downstream task transfer. In parallel, the software ecosystem around transformers is maturing rapidly, with libraries like HuggingFace and BertViz enabling streamlined training and introspection. In this work, we make one of the first attempts to systematically evaluate transformers on molecular property prediction tasks via our ChemBERTa model. ChemBERTa scales well with pretraining dataset size, offering competitive downstream performance on MoleculeNet and useful attention-based visualization modalities. Our results suggest that transformers offer a promising avenue of future work for molecular representation learning and property prediction. To facilitate these efforts, we release a curated dataset of 77M SMILES from PubChem suitable for large-scale self-supervised pretraining.

Submitted to NeurIPS 2020 ML for Molecules Workshop

References in corpus (8)

• Semi-Supervised Classification with Graph Convolutional Networks
• Neural Message Passing for Quantum Chemistry
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Molecular Graph Convolutions: Moving Beyond Fingerprints
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• SMILES Transformer: Pre-trained Molecular Fingerprint for Low Data Drug Discovery
• Quantifying the Carbon Emissions of Machine Learning
• Molecule Attention Transformer

Cited by in corpus (28)

• Molecular Contrastive Learning of Representations via Graph Neural Networks
• polyBERT: A chemical language model to enable fully machine-driven ultrafast polymer informatics
• Transformers in Healthcare: A Survey
• Materials science in the era of large language models: a perspective
• Improving Molecular Contrastive Learning via Faulty Negative Mitigation and Decomposed Fragment Contrast
• Empowering Molecule Discovery for Molecule-Caption Translation with Large Language Models: A ChatGPT Perspective
• Learning from flowsheets: A generative transformer model for autocompletion of flowsheets
• Chemical-Reaction-Aware Molecule Representation Learning
• Artificial Intelligence in Materials Science and Engineering: Current Landscape, Key Challenges, and Future Trajectorie
• Multimodal Model with Text and Drug Embeddings for Adverse Drug Reaction Classification
• Prot2Text: Multimodal Protein's Function Generation with GNNs and Transformers
• Advancing bioinformatics with large language models: components, applications and perspectives
• Learn molecular representations from large-scale unlabeled molecules for drug discovery
• Spiers Memorial Lecture: How to do impactful research in artificial intelligence for chemistry and materials science
• Machine Learning Based Prediction of Proton Conductivity in Metal-Organic Frameworks
• Dual-view Molecule Pre-training
• Multimodal machine learning with large language embedding model for polymer property prediction
• BatmanNet: Bi-branch Masked Graph Transformer Autoencoder for Molecular Representation
• Pre-Trained Language Models for Keyphrase Prediction: A Review
• Prompt Engineering for Transformer-based Chemical Similarity Search Identifies Structurally Distinct Functional Analogues
• Scalable Training of Trustworthy and Energy-Efficient Predictive Graph Foundation Models for Atomistic Materials Modeling: A Case Study with HydraGNN
• Differentiable Physics: A Position Piece
• IMG2SMI: Translating Molecular Structure Images to Simplified Molecular-input Line-entry System
• Artificial Intelligence in Drug Discovery: Applications and Techniques
• Predicting Drug-Drug Interactions Using Heterogeneous Graph Neural Networks: HGNN-DDI
• A Bayesian Flow Network Framework for Chemistry Tasks
• Tokenization for Molecular Foundation Models
• Using Clinical Drug Representations for Improving Mortality and Length of Stay Predictions