Efficient generative modeling of protein sequences using simple autoregressive models

Abstract

Generative models emerge as promising candidates for novel sequence-data driven approaches to protein design, and for the extraction of structural and functional information about proteins deeply hidden in rapidly growing sequence databases. Here we propose simple autoregressive models as highly accurate but computationally efficient generative sequence models. We show that they perform similarly to existing approaches based on Boltzmann machines or deep generative models, but at a substantially lower computational cost (by a factor between $10^2$ and $10^3$). Furthermore, the simple structure of our models has distinctive mathematical advantages, which translate into an improved applicability in sequence generation and evaluation. Within these models, we can easily estimate both the probability of a given sequence, and, using the model's entropy, the size of the functional sequence space related to a specific protein family. In the example of response regulators, we find a huge number of ca. $10^{68}$ possible sequences, which nevertheless constitute only the astronomically small fraction $10^{-80}$ of all amino-acid sequences of the same length. These findings illustrate the potential and the difficulty in exploring sequence space via generative sequence models.

12 pages, 4 Figures + Supplementary Material

References in corpus (4)

• Accurate De Novo Prediction of Protein Contact Map by Ultra-Deep Learning Model
• Improved contact prediction in proteins: Using pseudolikelihoods to infer Potts models
• Sparse generative modeling via parameter-reduction of Boltzmann machines: application to protein-sequence families
• Modeling sequence-space exploration and emergence of epistatic signals in protein evolution

Cited by in corpus (17)

• Epistatic models predict mutable sites in SARS-CoV-2 proteins and epitopes
• Generative Capacity of Probabilistic Protein Sequence Models
• Computational protein design with evolutionary-based and physics-inspired modeling: current and future synergies
• Machine-learning-assisted Monte Carlo fails at sampling computationally hard problems
• Sampling with flows, diffusion and autoregressive neural networks: A spin-glass perspective
• A Bayesian generative neural network framework for epidemic inference problems
• Is novelty predictable?
• Correlations from structure and phylogeny combine constructively in the inference of protein partners from sequences
• Introduction to the dynamics of disordered systems: equilibrium and gradient descent
• Towards Parsimonious Generative Modeling of RNA Families
• Phylogenetic Corrections and Higher-Order Sequence Statistics in Protein Families: The Potts Model vs MSA Transformer
• Inferring Higher-Order Couplings with Neural Networks
• SMCLM: Semantically Meaningful Causal Language Modeling for Autoregressive Paraphrase Generation
• Nearest-Neighbours Neural Network architecture for efficient sampling of statistical physics models
• Small Coupling Expansion for Multiple Sequence Alignment
• Functional bottlenecks can emerge from non-epistatic underlying traits
• Pseudo-likelihood produces associative memories able to generalize, even for asymmetric couplings