Publicly Available Clinical BERT Embeddings

Abstract

Contextual word embedding models such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2018) have dramatically improved performance for many natural language processing (NLP) tasks in recent months. However, these models have been minimally explored on specialty corpora, such as clinical text; moreover, in the clinical domain, no publicly-available pre-trained BERT models yet exist. In this work, we address this need by exploring and releasing BERT models for clinical text: one for generic clinical text and another for discharge summaries specifically. We demonstrate that using a domain-specific model yields performance improvements on three common clinical NLP tasks as compared to nonspecific embeddings. These domain-specific models are not as performant on two clinical de-identification tasks, and argue that this is a natural consequence of the differences between de-identified source text and synthetically non de-identified task text.

Clinical Natural Language Processing (ClinicalNLP) Workshop at NAACL 2019

Cited by in corpus (11)

• Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction
• MT-BioNER: Multi-task Learning for Biomedical Named Entity Recognition using Deep Bidirectional Transformers
• Progress Notes Classification and Keyword Extraction using Attention-based Deep Learning Models with BERT
• Automated Labelling using an Attention model for Radiology reports of MRI scans (ALARM)
• CheXpert++: Approximating the CheXpert labeler for Speed,Differentiability, and Probabilistic Output
• Biomedical Entity Representations with Synonym Marginalization
• A Comparison of Pre-trained Vision-and-Language Models for Multimodal Representation Learning across Medical Images and Reports
• Application of Pre-training Models in Named Entity Recognition
• Robustly Pre-trained Neural Model for Direct Temporal Relation Extraction
• COMPOSE: Cross-Modal Pseudo-Siamese Network for Patient Trial Matching
• Prediction of ICD Codes with Clinical BERT Embeddings and Text Augmentation with Label Balancing using MIMIC-III