TransPolymer: a Transformer-based language model for polymer property predictions

Abstract

Accurate and efficient prediction of polymer properties is of great significance in polymer design. Conventionally, expensive and time-consuming experiments or simulations are required to evaluate polymer functions. Recently, Transformer models, equipped with self-attention mechanisms, have exhibited superior performance in natural language processing. However, such methods have not been investigated in polymer sciences. Herein, we report TransPolymer, a Transformer-based language model for polymer property prediction. Our proposed polymer tokenizer with chemical awareness enables learning representations from polymer sequences. Rigorous experiments on ten polymer property prediction benchmarks demonstrate the superior performance of TransPolymer. Moreover, we show that TransPolymer benefits from pretraining on large unlabeled dataset via Masked Language Modeling. Experimental results further manifest the important role of self-attention in modeling polymer sequences. We highlight this model as a promising computational tool for promoting rational polymer design and understanding structure-property relationships from a data science view.

References in corpus (6)

• Polymer Informatics: Current Status and Critical Next Steps
• UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
• Keeping it Simple: Language Models can learn Complex Molecular Distributions
• Polymer Informatics with Multi-Task Learning
• Orbital Graph Convolutional Neural Network for Material Property Prediction
• A graph representation of molecular ensembles for polymer property prediction

Cited by in corpus (8)

• Bidirectional Generation of Structure and Properties Through a Single Molecular Foundation Model
• Artificial Intelligence in Materials Science and Engineering: Current Landscape, Key Challenges, and Future Trajectorie
• Tutorial: AI-assisted exploration and active design of polymers with high intrinsic thermal conductivity
• Multimodal machine learning with large language embedding model for polymer property prediction
• Toward Sustainable Polymer Design: A Molecular Dynamics-Informed Machine Learning Approach for Vitrimers
• Evaluating the Performance and Robustness of LLMs in Materials Science Q&A and Property Predictions
• Tokenization for Molecular Foundation Models
• An Encoder-Decoder Foundation Chemical Language Model for Generative Polymer Design