Evaluating Protein Transfer Learning with TAPE

Abstract

Protein modeling is an increasingly popular area of machine learning research. Semi-supervised learning has emerged as an important paradigm in protein modeling due to the high cost of acquiring supervised protein labels, but the current literature is fragmented when it comes to datasets and standardized evaluation techniques. To facilitate progress in this field, we introduce the Tasks Assessing Protein Embeddings (TAPE), a set of five biologically relevant semi-supervised learning tasks spread across different domains of protein biology. We curate tasks into specific training, validation, and test splits to ensure that each task tests biologically relevant generalization that transfers to real-life scenarios. We benchmark a range of approaches to semi-supervised protein representation learning, which span recent work as well as canonical sequence learning techniques. We find that self-supervised pretraining is helpful for almost all models on all tasks, more than doubling performance in some cases. Despite this increase, in several cases features learned by self-supervised pretraining still lag behind features extracted by state-of-the-art non-neural techniques. This gap in performance suggests a huge opportunity for innovative architecture design and improved modeling paradigms that better capture the signal in biological sequences. TAPE will help the machine learning community focus effort on scientifically relevant problems. Toward this end, all data and code used to run these experiments are available at https://github.com/songlab-cal/tape.

20 pages, 4 figures

References in corpus (4)

• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• Training Deep Nets with Sublinear Memory Cost
• Learning protein sequence embeddings using information from structure
• Multiplicative LSTM for sequence modelling

Cited by in corpus (20)

• Pretrained Transformers as Universal Computation Engines
• ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing
• AdaLead: A simple and robust adaptive greedy search algorithm for sequence design
• CogMol: Target-Specific and Selective Drug Design for COVID-19 Using Deep Generative Models
• Pre-Training of Deep Bidirectional Protein Sequence Representations with Structural Information
• Modeling Protein Using Large-scale Pretrain Language Model
• Deep Learning in Protein Structural Modeling and Design
• Blockwise Self-Attention for Long Document Understanding
• A Novel Framework Integrating AI Model and Enzymological Experiments Promotes Identification of SARS-CoV-2 3CL Protease Inhibitors and Activity-based Probe
• A primer on model-guided exploration of fitness landscapes for biological sequence design
• Dual-view Molecule Pre-training
• Profile Prediction: An Alignment-Based Pre-Training Task for Protein Sequence Models
• Pre-training Co-evolutionary Protein Representation via A Pairwise Masked Language Model
• Noise-Robust End-to-End Quantum Control using Deep Autoregressive Policy Networks
• Evolution Is All You Need: Phylogenetic Augmentation for Contrastive Learning
• Pre-training Protein Language Models with Label-Agnostic Binding Pairs Enhances Performance in Downstream Tasks
• PaccMann${}^{RL}$ on SARS-CoV-2: Designing antiviral candidates with conditional generative models
• Fixed-Length Protein Embeddings using Contextual Lenses
• Adaptive Residue-wise Profile Fusion for Low Homologous Protein SecondaryStructure Prediction Using External Knowledge
• Don't Sweep your Learning Rate under the Rug: A Closer Look at Cross-modal Transfer of Pretrained Transformers