A Large Encoder-Decoder Family of Foundation Models For Chemical Language

Abstract

Large-scale pre-training methodologies for chemical language models represent a breakthrough in cheminformatics. These methods excel in tasks such as property prediction and molecule generation by learning contextualized representations of input tokens through self-supervised learning on large unlabeled corpora. Typically, this involves pre-training on unlabeled data followed by fine-tuning on specific tasks, reducing dependence on annotated datasets and broadening chemical language representation understanding. This paper introduces a large encoder-decoder chemical foundation models pre-trained on a curated dataset of 91 million SMILES samples sourced from PubChem, which is equivalent to 4 billion of molecular tokens. The proposed foundation model supports different complex tasks, including quantum property prediction, and offer flexibility with two main variants (289M and $8\times 289M$). Our experiments across multiple benchmark datasets validate the capacity of the proposed model in providing state-of-the-art results for different tasks. We also provide a preliminary assessment of the compositionality of the embedding space as a prerequisite for the reasoning tasks. We demonstrate that the produced latent space is separable compared to the state-of-the-art with few-shot learning capabilities.

14 pages, 3 figures, 14 tables

References in corpus (15)

• Molecular Transformer - A Model for Uncertainty-Calibrated Chemical Reaction Prediction
• Molecular Contrastive Learning of Representations via Graph Neural Networks
• ChemRL-GEM: Geometry Enhanced Molecular Representation Learning for Property Prediction
• Self-Supervised Graph Transformer on Large-Scale Molecular Data
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Keeping it Simple: Language Models can learn Complex Molecular Distributions
• Exploring Chemical Space using Natural Language Processing Methodologies for Drug Discovery
• Pre-training Molecular Graph Representation with 3D Geometry
• What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks
• Bidirectional Generation of Structure and Properties Through a Single Molecular Foundation Model
• Foundation Models for Decision Making: Problems, Methods, and Opportunities
• Domain-Agnostic Molecular Generation with Chemical Feedback
• LIMO: Latent Inceptionism for Targeted Molecule Generation
• GP-MoLFormer: A Foundation Model For Molecular Generation

Cited by in corpus (1)

• Digital Surfactant