Probing Biomedical Embeddings from Language Models

Abstract

Contextualized word embeddings derived from pre-trained language models (LMs) show significant improvements on downstream NLP tasks. Pre-training on domain-specific corpora, such as biomedical articles, further improves their performance. In this paper, we conduct probing experiments to determine what additional information is carried intrinsically by the in-domain trained contextualized embeddings. For this we use the pre-trained LMs as fixed feature extractors and restrict the downstream task models to not have additional sequence modeling layers. We compare BERT, ELMo, BioBERT and BioELMo, a biomedical version of ELMo trained on 10M PubMed abstracts. Surprisingly, while fine-tuned BioBERT is better than BioELMo in biomedical NER and NLI tasks, as a fixed feature extractor BioELMo outperforms BioBERT in our probing tasks. We use visualization and nearest neighbor analysis to show that better encoding of entity-type and relational information leads to this superiority.

NAACL-HLT 2019 Workshop on Evaluating Vector Space Representations for NLP (RepEval)

References in corpus (7)

• Distributed Representations of Words and Phrases and their Compositionality
• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• Enhanced LSTM for Natural Language Inference
• A large annotated corpus for learning natural language inference
• Attending to Characters in Neural Sequence Labeling Models
• Clinical Concept Extraction with Contextual Word Embedding
• Lessons from Natural Language Inference in the Clinical Domain

Cited by in corpus (11)

• Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets
• Incorporating Domain Knowledge into Medical NLI using Knowledge Graphs
• Deep Contextualized Biomedical Abbreviation Expansion
• A Hybrid Approach to Measure Semantic Relatedness in Biomedical Concepts
• Stress Test Evaluation of Biomedical Word Embeddings
• Benchmarking BioRelEx for Entity Tagging and Relation Extraction
• Zero-Shot Clinical Acronym Expansion via Latent Meaning Cells
• Parsing Early Modern English for Linguistic Search
• Probing Pre-Trained Language Models for Disease Knowledge
• Recent Advances in Automated Question Answering In Biomedical Domain
• Let Your Heart Speak in its Mother Tongue: Multilingual Captioning of Cardiac Signals