Protein language models trained on multiple sequence alignments learn phylogenetic relationships

Abstract

Self-supervised neural language models with attention have recently been applied to biological sequence data, advancing structure, function and mutational effect prediction. Some protein language models, including MSA Transformer and AlphaFold's EvoFormer, take multiple sequence alignments (MSAs) of evolutionarily related proteins as inputs. Simple combinations of MSA Transformer's row attentions have led to state-of-the-art unsupervised structural contact prediction. We demonstrate that similarly simple, and universal, combinations of MSA Transformer's column attentions strongly correlate with Hamming distances between sequences in MSAs. Therefore, MSA-based language models encode detailed phylogenetic relationships. We further show that these models can separate coevolutionary signals encoding functional and structural constraints from phylogenetic correlations reflecting historical contingency. To assess this, we generate synthetic MSAs, either without or with phylogeny, from Potts models trained on natural MSAs. We find that unsupervised contact prediction is substantially more resilient to phylogenetic noise when using MSA Transformer versus inferred Potts models.

arXiv comment29 pages, 10 figures, 7 tables

References in corpus (9)

• Direct-coupling analysis of residue co-evolution captures native contacts across many protein families
• Identification of direct residue contacts in protein-protein interaction by message passing
• Improved contact prediction in proteins: Using pseudolikelihoods to infer Potts models
• Axial Attention in Multidimensional Transformers
• Inverse Statistical Physics of Protein Sequences: A Key Issues Review
• Fast pseudolikelihood maximization for direct-coupling analysis of protein structure from many homologous amino-acid sequences
• How pairwise coevolutionary models capture the collective residue variability in proteins
• Generative power of a protein language model trained on multiple sequence alignments
• Generative Capacity of Probabilistic Protein Sequence Models

Cited by in corpus (5)

• Generative power of a protein language model trained on multiple sequence alignments
• Impact of phylogeny on structural contact inference from protein sequence data
• Understanding the Natural Language of DNA using Encoder-Decoder Foundation Models with Byte-level Precision
• Phylogenetic Corrections and Higher-Order Sequence Statistics in Protein Families: The Potts Model vs MSA Transformer
• Impact of phylogeny on the inference of functional sectors from protein sequence data