ProGen: Language Modeling for Protein Generation
arXiv:2004.03497
Abstract
Generative modeling for protein engineering is key to solving fundamental problems in synthetic biology, medicine, and material science. We pose protein engineering as an unsupervised sequence generation problem in order to leverage the exponentially growing set of proteins that lack costly, structural annotations. We train a 1.2B-parameter language model, ProGen, on ~280M protein sequences conditioned on taxonomic and keyword tags such as molecular function and cellular component. This provides ProGen with an unprecedented range of evolutionary sequence diversity and allows it to generate with fine-grained control as demonstrated by metrics based on primary sequence similarity, secondary structure accuracy, and conformational energy.
References in corpus (8)
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- CTRL: A Conditional Transformer Language Model for Controllable Generation
- Machine learning-assisted directed protein evolution with combinatorial libraries
- Generating Long Sequences with Sparse Transformers
- Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
- Design of metalloproteins and novel protein folds using variational autoencoders
- Defending Against Neural Fake News
- How to Hallucinate Functional Proteins
Cited by in corpus (11)
- Rethinking Attention with Performers
- AdaLead: A simple and robust adaptive greedy search algorithm for sequence design
- Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers
- Combiner: Full Attention Transformer with Sparse Computation Cost
- A primer on model-guided exploration of fitness landscapes for biological sequence design
- Profile Prediction: An Alignment-Based Pre-Training Task for Protein Sequence Models
- Deep Extrapolation for Attribute-Enhanced Generation
- Pre-training Co-evolutionary Protein Representation via A Pairwise Masked Language Model
- Evolution Is All You Need: Phylogenetic Augmentation for Contrastive Learning
- On the Expressive Power of Self-Attention Matrices
- On Learning the Transformer Kernel