Publicly Available Clinical BERT Embeddings

Abstract

Contextual word embedding models such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2018) have dramatically improved performance for many natural language processing (NLP) tasks in recent months. However, these models have been minimally explored on specialty corpora, such as clinical text; moreover, in the clinical domain, no publicly-available pre-trained BERT models yet exist. In this work, we address this need by exploring and releasing BERT models for clinical text: one for generic clinical text and another for discharge summaries specifically. We demonstrate that using a domain-specific model yields performance improvements on three common clinical NLP tasks as compared to nonspecific embeddings. These domain-specific models are not as performant on two clinical de-identification tasks, and argue that this is a natural consequence of the differences between de-identified source text and synthetically non de-identified task text.

Clinical Natural Language Processing (ClinicalNLP) Workshop at NAACL 2019

References in corpus (5)

• Distributed Representations of Words and Phrases and their Compositionality
• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• Enhancing Clinical Concept Extraction with Contextual Embeddings
• Clinical Concept Extraction with Contextual Word Embedding
• A Deep Learning Architecture for De-identification of Patient Notes: Implementation and Evaluation

Cited by in corpus (75)

• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• Pre-trained Models for Natural Language Processing: A Survey
• Large AI Models in Health Informatics: Applications, Challenges, and the Future
• Drug Repurposing for COVID-19 via Knowledge Graph Completion
• Knowledge Matters: Radiology Report Generation with General and Specific Knowledge
• Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training
• A Comparative Study of Pretrained Language Models for Long Clinical Text
• Generative AI for Synthetic Data Across Multiple Medical Modalities: A Systematic Review of Recent Developments and Challenges
• Ontology-driven weak supervision for clinical entity classification in electronic health records
• CXR-CLIP: Toward Large Scale Chest X-ray Language-Image Pre-training
• Enhancing Phenotype Recognition in Clinical Notes Using Large Language Models: PhenoBCBERT and PhenoGPT
• Explainable Identification of Dementia from Transcripts using Transformer Networks
• Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction
• Hierarchical Label-wise Attention Transformer Model for Explainable ICD Coding
• SciBERT: A Pretrained Language Model for Scientific Text
• ERNIE-GeoL: A Geography-and-Language Pre-trained Model and its Applications in Baidu Maps
• Attention-based Clinical Note Summarization
• Pre-trained Language Models in Biomedical Domain: A Systematic Survey
• MT-BioNER: Multi-task Learning for Biomedical Named Entity Recognition using Deep Bidirectional Transformers
• Several categories of Large Language Models (LLMs): A Short Survey
• A Customized Text Sanitization Mechanism with Differential Privacy
• Progress Notes Classification and Keyword Extraction using Attention-based Deep Learning Models with BERT
• Hurtful Words: Quantifying Biases in Clinical Contextual Word Embeddings
• Two Directions for Clinical Data Generation with Large Language Models: Data-to-Label and Label-to-Data
• Training-free Lexical Backdoor Attacks on Language Models
• Deep Learning Approaches for Extracting Adverse Events and Indications of Dietary Supplements from Clinical Text
• Automated Labelling using an Attention model for Radiology reports of MRI scans (ALARM)
• Phenotyping of Clinical Notes with Improved Document Classification Models Using Contextualized Neural Language Models
• Read, Attend, and Code: Pushing the Limits of Medical Codes Prediction from Clinical Notes by Machines
• Testing the Generalization of Neural Language Models for COVID-19 Misinformation Detection
• Deep Learning Approach for Classifying the Aggressive Comments on Social Media: Machine Translated Data Vs Real Life Data
• Label Dependent Attention Model for Disease Risk Prediction Using Multimodal Electronic Health Records
• Recent Advances in Named Entity Recognition: A Comprehensive Survey and Comparative Study
• Application of Deep Learning in Generating Structured Radiology Reports: A Transformer-Based Technique
• A Deep Learning Pipeline for Patient Diagnosis Prediction Using Electronic Health Records
• CheXpert++: Approximating the CheXpert labeler for Speed,Differentiability, and Probabilistic Output
• CLIN-X: pre-trained language models and a study on cross-task transfer for concept extraction in the clinical domain
• Pre-Trained Language Models for Keyphrase Prediction: A Review
• MathBERT: A Pre-trained Language Model for General NLP Tasks in Mathematics Education
• Scientific Language Models for Biomedical Knowledge Base Completion: An Empirical Study
• Clinical Relation Extraction Using Transformer-based Models
• Multitask Recalibrated Aggregation Network for Medical Code Prediction
• A Meta-Evaluation of Faithfulness Metrics for Long-Form Hospital-Course Summarization
• Entity Recognition from Colloquial Text
• PromptLink: Leveraging Large Language Models for Cross-Source Biomedical Concept Linking
• Toward expanding the scope of radiology report summarization to multiple anatomies and modalities
• Extracting Adverse Drug Events from Clinical Notes
• Toward Improving Health Literacy in Patient Education Materials with Neural Machine Translation Models
• Biomedical Entity Representations with Synonym Marginalization
• Reviewing Clinical Knowledge in Medical Large Language Models: Training and Beyond
• SkillSpan: Hard and Soft Skill Extraction from English Job Postings
• A Comparison of Pre-trained Vision-and-Language Models for Multimodal Representation Learning across Medical Images and Reports
• A Hybrid Approach to Measure Semantic Relatedness in Biomedical Concepts
• Generative AI in Health Economics and Outcomes Research: A Taxonomy of Key Definitions and Emerging Applications, an ISPOR Working Group Report
• Application of Pre-training Models in Named Entity Recognition
• Robustly Pre-trained Neural Model for Direct Temporal Relation Extraction
• Discontinuous Named Entity Recognition as Maximal Clique Discovery
• Exploring the Impact of Corpus Diversity on Financial Pretrained Language Models
• Coarse-to-fine Knowledge Graph Domain Adaptation based on Distantly-supervised Iterative Training
• Explainable Health Risk Predictor with Transformer-based Medicare Claim Encoder
• Deep Insights into Cognitive Decline: A Survey of Leveraging Non-Intrusive Modalities with Deep Learning Techniques
• Quantifying Explainability in NLP and Analyzing Algorithms for Performance-Explainability Tradeoff
• Prediction of ICD Codes with Clinical BERT Embeddings and Text Augmentation with Label Balancing using MIMIC-III
• Zero-Shot Clinical Acronym Expansion via Latent Meaning Cells
• Ascle: A Python Natural Language Processing Toolkit for Medical Text Generation
• Differential Privacy for Text Analytics via Natural Text Sanitization
• COMPOSE: Cross-Modal Pseudo-Siamese Network for Patient Trial Matching
• Clinical Trial Information Extraction with BERT
• Priberam at MESINESP Multi-label Classification of Medical Texts Task
• #StayHome or #Marathon? Social Media Enhanced Pandemic Surveillance on Spatial-temporal Dynamic Graphs
• Discovering novel drug-supplement interactions using a dietary supplements knowledge graph generated from the biomedical literature
• Improving Clinical Outcome Predictions Using Convolution over Medical Entities with Multimodal Learning
• Discovering key topics from short, real-world medical inquiries via natural language processing and unsupervised learning
• Evaluating Biomedical BERT Models for Vocabulary Alignment at Scale in the UMLS Metathesaurus
• Don't be fooled: label leakage in explanation methods and the importance of their quantitative evaluation