Enhancing Clinical Concept Extraction with Contextual Embeddings

Abstract

Neural network-based representations ("embeddings") have dramatically advanced natural language processing (NLP) tasks, including clinical NLP tasks such as concept extraction. Recently, however, more advanced embedding methods and representations (e.g., ELMo, BERT) have further pushed the state-of-the-art in NLP, yet there are no common best practices for how to integrate these representations into clinical tasks. The purpose of this study, then, is to explore the space of possible options in utilizing these new models for clinical concept extraction, including comparing these to traditional word embedding methods (word2vec, GloVe, fastText). Both off-the-shelf open-domain embeddings and pre-trained clinical embeddings from MIMIC-III are evaluated. We explore a battery of embedding methods consisting of traditional word embeddings and contextual embeddings, and compare these on four concept extraction corpora: i2b2 2010, i2b2 2012, SemEval 2014, and SemEval 2015. We also analyze the impact of the pre-training time of a large language model like ELMo or BERT on the extraction performance. Last, we present an intuitive way to understand the semantic information encoded by contextual embeddings. Contextual embeddings pre-trained on a large clinical corpus achieves new state-of-the-art performances across all concept extraction tasks. The best-performing model outperforms all state-of-the-art methods with respective F1-measures of 90.25, 93.18 (partial), 80.74, and 81.65. We demonstrate the potential of contextual embeddings through the state-of-the-art performance these methods achieve on clinical concept extraction. Additionally, we demonstrate contextual embeddings encode valuable semantic information not accounted for in traditional word representations.

Journal of the American Medical Informatics Association

References in corpus (10)

• Distributed Representations of Words and Phrases and their Compositionality
• TensorFlow: A system for large-scale machine learning
• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• Bidirectional Attention Flow for Machine Comprehension
• ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
• Enriching Word Vectors with Subword Information
• Recurrent neural networks with specialized word embeddings for health-domain named-entity recognition
• Bidirectional LSTM-CRF for Clinical Concept Extraction
• Clinical Concept Extraction with Contextual Word Embedding
• Word Embedding based Correlation Model for Question/Answer Matching

Cited by in corpus (34)

• Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
• Publicly Available Clinical BERT Embeddings
• ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
• Clinical Concept Extraction: a Methodology Review
• Transformers in Healthcare: A Survey
• Ontology-driven weak supervision for clinical entity classification in electronic health records
• Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets
• BERT-based Ranking for Biomedical Entity Normalization
• Pre-trained Language Models in Biomedical Domain: A Systematic Survey
• Lightweight Transformers for Clinical Natural Language Processing
• Hurtful Words: Quantifying Biases in Clinical Contextual Word Embeddings
• End-to-end Named Entity Recognition and Relation Extraction using Pre-trained Language Models
• CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters
• Federated pretraining and fine tuning of BERT using clinical notes from multiple silos
• Application of Deep Learning in Generating Structured Radiology Reports: A Transformer-Based Technique
• Representation Learning for Electronic Health Records
• NEAR: Named Entity and Attribute Recognition of clinical concepts
• Human-interpretable clustering of short-text using large language models
• Extracting UMLS Concepts from Medical Text Using General and Domain-Specific Deep Learning Models
• Clinical Relation Extraction Using Transformer-based Models
• Domain-Specific Pretraining for Vertical Search: Case Study on Biomedical Literature
• Robustly Pre-trained Neural Model for Direct Temporal Relation Extraction
• Knowledge Guided Named Entity Recognition for BioMedical Text
• Towards Automated Anamnesis Summarization: BERT-based Models for Symptom Extraction
• Self-Supervised Contextual Language Representation of Radiology Reports to Improve the Identification of Communication Urgency
• Concept Identification of Directly and Indirectly Related Mentions Referring to Groups of Persons
• CancerBERT: a BERT model for Extracting Breast Cancer Phenotypes from Electronic Health Records
• Towards NLP-supported Semantic Data Management
• Leveraging Spatial Information in Radiology Reports for Ischemic Stroke Phenotyping
• RadLex Normalization in Radiology Reports
• Extracting clinical concepts from user queries
• Named Entity Normalization Model Using Edge Weight Updating Neural Network: Assimilation Between Knowledge-Driven Graph and Data-Driven Graph
• Robust Benchmarking for Machine Learning of Clinical Entity Extraction
• JaMIE: A Pipeline Japanese Medical Information Extraction System