SMILES Transformer: Pre-trained Molecular Fingerprint for Low Data Drug Discovery

Abstract

In drug-discovery-related tasks such as virtual screening, machine learning is emerging as a promising way to predict molecular properties. Conventionally, molecular fingerprints (numerical representations of molecules) are calculated through rule-based algorithms that map molecules to a sparse discrete space. However, these algorithms perform poorly for shallow prediction models or small datasets. To address this issue, we present SMILES Transformer. Inspired by Transformer and pre-trained language models from natural language processing, SMILES Transformer learns molecular fingerprints through unsupervised pre-training of the sequence-to-sequence language model using a huge corpus of SMILES, a text representation system for molecules. We performed benchmarks on 10 datasets against existing fingerprints and graph-based methods and demonstrated the superiority of the proposed algorithms in small-data settings where pre-training facilitated good generalization. Moreover, we define a novel metric to concurrently measure model accuracy and data efficiency.

9 pages, 4 figures

References in corpus (2)

• Sequence to Sequence Learning with Neural Networks
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation

Cited by in corpus (8)

• Chemical-Reaction-Aware Molecule Representation Learning
• Artificial Intelligence in Materials Science and Engineering: Current Landscape, Key Challenges, and Future Trajectorie
• Learn molecular representations from large-scale unlabeled molecules for drug discovery
• Dual-view Molecule Pre-training
• Learning Attributed Graph Representations with Communicative Message Passing Transformer
• Artificial Intelligence in Drug Discovery: Applications and Techniques
• Distributed Deep Learning in Open Collaborations
• Using Clinical Drug Representations for Improving Mortality and Length of Stay Predictions