ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission

Abstract

Clinical notes contain information about patients that goes beyond structured data like lab values and medications. However, clinical notes have been underused relative to structured data, because notes are high-dimensional and sparse. This work develops and evaluates representations of clinical notes using bidirectional transformers (ClinicalBERT). ClinicalBERT uncovers high-quality relationships between medical concepts as judged by humans. ClinicalBert outperforms baselines on 30-day hospital readmission prediction using both discharge summaries and the first few days of notes in the intensive care unit. Code and model parameters are available.

CHIL 2020 Workshop

References in corpus (4)

• Distributed Representations of Words and Phrases and their Compositionality
• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• Deep EHR: A Survey of Recent Advances in Deep Learning Techniques for Electronic Health Record (EHR) Analysis
• Enhancing Clinical Concept Extraction with Contextual Embeddings

Cited by in corpus (79)

• A Survey on Visual Transformer
• Pre-trained Models for Natural Language Processing: A Survey
• Large language models in medicine: the potentials and pitfalls
• Enhancing Clinical Concept Extraction with Contextual Embeddings
• Natural Language Processing for Smart Healthcare
• FinBERT: A Pretrained Language Model for Financial Communications
• Transformers in Healthcare: A Survey
• Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
• A Comparative Study of Pretrained Language Models for Long Clinical Text
• Biomedical Question Answering: A Survey of Approaches and Challenges
• Does the Magic of BERT Apply to Medical Code Assignment? A Quantitative Study
• Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction
• Hierarchical Label-wise Attention Transformer Model for Explainable ICD Coding
• SciBERT: A Pretrained Language Model for Scientific Text
• ERNIE-GeoL: A Geography-and-Language Pre-trained Model and its Applications in Baidu Maps
• Automated Quality Assessment of Cognitive Behavioral Therapy Sessions Through Highly Contextualized Language Representations
• A Unified Review of Deep Learning for Automated Medical Coding
• Pre-trained Language Models in Biomedical Domain: A Systematic Survey
• MT-BioNER: Multi-task Learning for Biomedical Named Entity Recognition using Deep Bidirectional Transformers
• CrisisTransformers: Pre-trained language models and sentence encoders for crisis-related social media texts
• Clinical Insights: A Comprehensive Review of Language Models in Medicine
• Hurtful Words: Quantifying Biases in Clinical Contextual Word Embeddings
• Two Directions for Clinical Data Generation with Large Language Models: Data-to-Label and Label-to-Data
• A Dual-Attention Learning Network with Word and Sentence Embedding for Medical Visual Question Answering
• Intimate Partner Violence and Injury Prediction From Radiology Reports
• E-BERT: A Phrase and Product Knowledge Enhanced Language Model for E-commerce
• Label Dependent Attention Model for Disease Risk Prediction Using Multimodal Electronic Health Records
• A Deep Learning Pipeline for Patient Diagnosis Prediction Using Electronic Health Records
• Representation Learning for Electronic Health Records
• Clinical XLNet: Modeling Sequential Clinical Notes and Predicting Prolonged Mechanical Ventilation
• CSS-LM: A Contrastive Framework for Semi-supervised Fine-tuning of Pre-trained Language Models
• Fast, Structured Clinical Documentation via Contextual Autocomplete
• MathBERT: A Pre-trained Language Model for General NLP Tasks in Mathematics Education
• Clinical Relation Extraction Using Transformer-based Models
• PharmMT: A Neural Machine Translation Approach to Simplify Prescription Directions
• A Scoping Review of Natural Language Processing in Addressing Medically Inaccurate Information: Errors, Misinformation, and Hallucination
• Domain-Specific Pretraining for Vertical Search: Case Study on Biomedical Literature
• BERTSurv: BERT-Based Survival Models for Predicting Outcomes of Trauma Patients
• BioMegatron: Larger Biomedical Domain Language Model
• Improving Dialogue Breakdown Detection with Semi-Supervised Learning
• MS2: Multi-Document Summarization of Medical Studies
• Towards Debiasing Sentence Representations
• Cross-Global Attention Graph Kernel Network Prediction of Drug Prescription
• Improving Early Sepsis Prediction with Multi Modal Learning
• Students Need More Attention: BERT-based AttentionModel for Small Data with Application to AutomaticPatient Message Triage
• Domain-Relevant Embeddings for Medical Question Similarity
• Train No Evil: Selective Masking for Task-Guided Pre-Training
• Application of Pre-training Models in Named Entity Recognition
• Interpretable bias mitigation for textual data: Reducing gender bias in patient notes while maintaining classification performance
• Effective Transfer Learning for Identifying Similar Questions: Matching User Questions to COVID-19 FAQs
• Query-Focused EHR Summarization to Aid Imaging Diagnosis
• An Interpretable End-to-end Fine-tuning Approach for Long Clinical Text
• Multi-Task Prediction of Clinical Outcomes in the Intensive Care Unit using Flexible Multimodal Transformers
• BERT-XML: Large Scale Automated ICD Coding Using BERT Pretraining
• Pre-training Language Model Incorporating Domain-specific Heterogeneous Knowledge into A Unified Representation
• Advances in Multi-turn Dialogue Comprehension: A Survey
• Leveraging Time Irreversibility with Order-Contrastive Pre-training
• Zero-Shot Clinical Acronym Expansion via Latent Meaning Cells
• Cost-effective Selection of Pretraining Data: A Case Study of Pretraining BERT on Social Media
• Machine Learning Applications for Therapeutic Tasks with Genomics Data
• Noise Pollution in Hospital Readmission Prediction: Long Document Classification with Reinforcement Learning
• Patient Outcome and Zero-shot Diagnosis Prediction with Hypernetwork-guided Multitask Learning
• VICCA: Visual Interpretation and Comprehension of Chest X-ray Anomalies in Generated Report Without Human Feedback
• Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Media
• Multilingual Medical Question Answering and Information Retrieval for Rural Health Intelligence Access
• SMedBERT: A Knowledge-Enhanced Pre-trained Language Model with Structured Semantics for Medical Text Mining
• Efficient Domain Adaptation of Language Models via Adaptive Tokenization
• Leveraging Pretrained Models for Automatic Summarization of Doctor-Patient Conversations
• Variational Knowledge Distillation for Disease Classification in Chest X-Rays
• Enriched Annotations for Tumor Attribute Classification from Pathology Reports with Limited Labeled Data
• Improving Clinical Outcome Predictions Using Convolution over Medical Entities with Multimodal Learning
• On the Universality of Deep Contextual Language Models
• Predicting Unplanned Readmissions with Highly Unstructured Data
• Don't be fooled: label leakage in explanation methods and the importance of their quantitative evaluation
• Let Your Heart Speak in its Mother Tongue: Multilingual Captioning of Cardiac Signals
• How to Leverage Multimodal EHR Data for Better Medical Predictions?
• Zero-shot Task Transfer for Invoice Extraction via Class-aware QA Ensemble
• Transformer Query-Target Knowledge Discovery (TEND): Drug Discovery from CORD-19
• Cascade Neural Ensemble for Identifying Scientifically Sound Articles