Molecular representation learning with language models and domain-relevant auxiliary tasks

Abstract

We apply a Transformer architecture, specifically BERT, to learn flexible and high quality molecular representations for drug discovery problems. We study the impact of using different combinations of self-supervised tasks for pre-training, and present our results for the established Virtual Screening and QSAR benchmarks. We show that: i) The selection of appropriate self-supervised task(s) for pre-training has a significant impact on performance in subsequent downstream tasks such as Virtual Screening. ii) Using auxiliary tasks with more domain relevance for Chemistry, such as learning to predict calculated molecular properties, increases the fidelity of our learnt representations. iii) Finally, we show that molecular representations learnt by our model `MolBert' improve upon the current state of the art on the benchmark datasets.

References in corpus (6)

• Adam: A Method for Stochastic Optimization
• Scikit-learn: Machine Learning in Python
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Improving Chemical Autoencoder Latent Space and Molecular De novo Generation Diversity with Heteroencoders
• Molecule Attention Transformer
• Learning to SMILE(S)

Cited by in corpus (8)

• Regression Transformer: Concurrent sequence regression and generation for molecular language modeling
• Improving Molecular Contrastive Learning via Faulty Negative Mitigation and Decomposed Fragment Contrast
• Chemical-Reaction-Aware Molecule Representation Learning
• Multimodal Model with Text and Drug Embeddings for Adverse Drug Reaction Classification
• Prot2Text: Multimodal Protein's Function Generation with GNNs and Transformers
• Dual-view Molecule Pre-training
• Artificial Intelligence in Drug Discovery: Applications and Techniques
• Predicting small molecules solubilities on endpoint devices using deep ensemble neural networks