Transformers and the representation of biomedical background knowledge

Abstract

Specialised transformers-based models (such as BioBERT and BioMegatron) are adapted for the biomedical domain based on publicly available biomedical corpora. As such, they have the potential to encode large-scale biological knowledge. We investigate the encoding and representation of biological knowledge in these models, and its potential utility to support inference in cancer precision medicine - namely, the interpretation of the clinical significance of genomic alterations. We compare the performance of different transformer baselines; we use probing to determine the consistency of encodings for distinct entities; and we use clustering methods to compare and contrast the internal properties of the embeddings for genes, variants, drugs and diseases. We show that these models do indeed encode biological knowledge, although some of this is lost in fine-tuning for specific tasks. Finally, we analyse how the models behave with regard to biases and imbalances in the dataset.

arXiv comment25 pages, 12 figures, supplementary methods, tables and figures at the end of the manuscript

References in corpus (5)

• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• Accelerated Hierarchical Density Clustering
• Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks
• Pre-trained Language Models in Biomedical Domain: A Systematic Survey
• Do Transformers Encode a Foundational Ontology? Probing Abstract Classes in Natural Language

Cited by in corpus (1)

• Large Language Models, scientific knowledge and factuality: A framework to streamline human expert evaluation