Is BERT a Cross-Disciplinary Knowledge Learner? A Surprising Finding of Pre-trained Models' Transferability
arXiv:2103.07162
Abstract
This paper investigates whether the power of the models pre-trained on text data, such as BERT, can be transferred to general token sequence classification applications. To verify pre-trained models' transferability, we test the pre-trained models on text classification tasks with meanings of tokens mismatches, and real-world non-text token sequence classification data, including amino acid, DNA, and music. We find that even on non-text data, the models pre-trained on text converge faster, perform better than the randomly initialized models, and only slightly worse than the models using task-specific knowledge. We also find that the representations of the text and non-text pre-trained models share non-trivial similarities.
arXiv commentFindings of EMNLP 2021
References in corpus (5)
- BioBERT: a pre-trained biomedical language representation model for biomedical text mining
- Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
- Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset
- An image representation based convolutional network for DNA classification
- Pre-Training of Deep Bidirectional Protein Sequence Representations with Structural Information