Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction

Abstract

Deep learning (DL) based predictive models from electronic health records (EHR) deliver impressive performance in many clinical tasks. Large training cohorts, however, are often required to achieve high accuracy, hindering the adoption of DL-based models in scenarios with limited training data size. Recently, bidirectional encoder representations from transformers (BERT) and related models have achieved tremendous successes in the natural language processing domain. The pre-training of BERT on a very large training corpus generates contextualized embeddings that can boost the performance of models trained on smaller datasets. We propose Med-BERT, which adapts the BERT framework for pre-training contextualized embedding models on structured diagnosis data from 28,490,650 patients EHR dataset. Fine-tuning experiments are conducted on two disease-prediction tasks: (1) prediction of heart failure in patients with diabetes and (2) prediction of pancreatic cancer from two clinical databases. Med-BERT substantially improves prediction accuracy, boosting the area under receiver operating characteristics curve (AUC) by 2.02-7.12%. In particular, pre-trained Med-BERT substantially improves the performance of tasks with very small fine-tuning training sets (300-500 samples) boosting the AUC by more than 20% or equivalent to the AUC of 10 times larger training set. We believe that Med-BERT will benefit disease-prediction studies with small local training datasets, reduce data collection expenses, and accelerate the pace of artificial intelligence aided healthcare.

L.R., X.Y., and Z.X. share first authorship of this work

References in corpus (15)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• A Simple Framework for Contrastive Learning of Visual Representations
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Scalable and accurate deep learning for electronic health records
• Predicting Cardiovascular Risk Factors from Retinal Fundus Photographs using Deep Learning
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Publicly Available Clinical BERT Embeddings
• ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
• Dipole: Diagnosis Prediction in Healthcare via Attention-based Bidirectional Recurrent Neural Networks
• Predicting Alzheimer's disease: a neuroimaging study with 3D convolutional neural networks
• How much data is needed to train a medical image deep learning system to achieve necessary high accuracy?
• RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism
• DocBERT: BERT for Document Classification

Cited by in corpus (10)

• Deep Representation Learning of Patient Data from Electronic Health Records (EHR): A Systematic Review
• Pre-trained Language Models in Biomedical Domain: A Systematic Survey
• Hi-BEHRT: Hierarchical Transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health records
• Accenture at CheckThat! 2020: If you say so: Post-hoc fact-checking of claims using transformer-based models
• BERTSurv: BERT-Based Survival Models for Predicting Outcomes of Trauma Patients
• Generalized and Transferable Patient Language Representation for Phenotyping with Limited Data
• SINA-BERT: A pre-trained Language Model for Analysis of Medical Texts in Persian
• MedGPT: Medical Concept Prediction from Clinical Narratives
• Transfer Learning in Electronic Health Records through Clinical Concept Embedding
• Explainable Health Risk Predictor with Transformer-based Medicare Claim Encoder