Evolution Is All You Need: Phylogenetic Augmentation for Contrastive Learning

Abstract

Self-supervised representation learning of biological sequence embeddings alleviates computational resource constraints on downstream tasks while circumventing expensive experimental label acquisition. However, existing methods mostly borrow directly from large language models designed for NLP, rather than with bioinformatics philosophies in mind. Recently, contrastive mutual information maximization methods have achieved state-of-the-art representations for ImageNet. In this perspective piece, we discuss how viewing evolution as natural sequence augmentation and maximizing information across phylogenetic "noisy channels" is a biologically and theoretically desirable objective for pretraining encoders. We first provide a review of current contrastive learning literature, then provide an illustrative example where we show that contrastive learning using evolutionary augmentation can be used as a representation learning objective which maximizes the mutual information between biological sequences and their conserved function, and finally outline rationale for this approach.

Machine Learning in Computational Biology (MLCB) 2020

References in corpus (17)

• A Simple Framework for Contrastive Learning of Visual Representations
• Momentum Contrast for Unsupervised Visual Representation Learning
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• What Makes for Good Views for Contrastive Learning?
• Evolution of Biological Complexity
• Contrastive Multiview Coding
• Deep Variational Information Bottleneck
• Learning Discrete Representations via Information Maximizing Self-Augmented Training
• On Mutual Information Maximization for Representation Learning
• Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
• Supervised Contrastive Learning
• A Mutual Information Maximization Perspective of Language Representation Learning
• Evaluating Protein Transfer Learning with TAPE
• ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing
• ProGen: Language Modeling for Protein Generation
• Visual Representations: Defining Properties and Deep Approximations