Regression Transformer: Concurrent sequence regression and generation for molecular language modeling

Abstract

Despite significant progress of generative models in the natural sciences, their controllability remains challenging. One fundamentally missing aspect of molecular or protein generative models is an inductive bias that can reflect continuous properties of interest. To that end, we propose the Regression Transformer (RT), a novel method that abstracts regression as a conditional sequence modeling problem. This introduces a new paradigm of multitask language models which seamlessly bridge sequence regression and conditional sequence generation. We thoroughly demonstrate that, despite using a nominal-scale training objective, the RT matches or surpasses the performance of conventional regression models in property prediction tasks of small molecules, proteins and chemical reactions. Critically, priming the same model with continuous properties yields a highly competitive conditional generative model that outperforms specialized approaches in a substructure-constrained, property-driven molecule generation benchmark. Our dichotomous approach is facilitated by a novel, alternating training scheme that enables the model to decorate seed sequences by desired properties, e.g., to optimize reaction yield. In sum, the RT is the first report of a multitask model that concurrently excels at predictive and generative tasks in biochemistry. This finds particular application in property-driven, local exploration of the chemical or protein space and could pave the road toward foundation models in material design. The code to reproduce all experiments of the paper is available at: https://github.com/IBM/regression-transformer

Updated paper, under review; Preliminary version as spotlight talk at ICLR 2022 workshop on Machine Learning for Drug Discovery

References in corpus (16)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Deep learning for molecular design - a review of the state of the art
• Multitask Prompted Training Enables Zero-Shot Task Generalization
• Decision Transformer: Reinforcement Learning via Sequence Modeling
• MoFlow: An Invertible Flow Model for Generating Molecular Graphs
• GraphAF: a Flow-based Autoregressive Model for Molecular Graph Generation
• GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generation
• Protein sequence design with deep generative models
• InCoder: A Generative Model for Code Infilling and Synthesis
• Gradio: Hassle-Free Sharing and Testing of ML Models in the Wild
• Molecular representation learning with language models and domain-relevant auxiliary tasks
• A Deep Generative Model for Molecule Optimization via One Fragment Modification
• Deep Molecular Dreaming: Inverse machine learning for de-novo molecular design and interpretability with surjective representations
• Efficient Training of Language Models to Fill in the Middle
• Learning to Extend Molecular Scaffolds with Structural Motifs
• Robust Multi-Objective Bayesian Optimization Under Input Noise

Cited by in corpus (7)

• Generative Language Model for Catalyst Discovery
• Accelerating Material Design with the Generative Toolkit for Scientific Discovery
• A Data-Driven Framework for Digital Transformation in Smart Cities: Integrating AI, Dashboards, and IoT Readiness
• Language models in molecular discovery
• Evolutionary Monte Carlo of QM properties in chemical space: Electrolyte design
• Integrating the Expected Future in Load Forecasts with Contextually Enhanced Transformer Models
• Generative Artificial Intelligence in Bioinformatics: A Systematic Review of Models, Applications, and Methodological Advances