Improved Hierarchical Patient Classification with Language Model Pretraining over Clinical Notes

Abstract

Clinical notes in electronic health records contain highly heterogeneous writing styles, including non-standard terminology or abbreviations. Using these notes in predictive modeling has traditionally required preprocessing (e.g. taking frequent terms or topic modeling) that removes much of the richness of the source data. We propose a pretrained hierarchical recurrent neural network model that parses minimally processed clinical notes in an intuitive fashion, and show that it improves performance for discharge diagnosis classification tasks on the Medical Information Mart for Intensive Care III (MIMIC-III) dataset, compared to models that treat the notes as an unordered collection of terms or that conduct no pretraining. We also apply an attribution technique to examples to identify the words that the model uses to make its prediction, and show the importance of the words' nearby context.

Machine Learning for Health (ML4H) at NeurIPS 2019 - extended abstract

References in corpus (14)

• Scikit-learn: Machine Learning in Python
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Axiomatic Attribution for Deep Networks
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Deep EHR: A Survey of Recent Advances in Deep Learning Techniques for Electronic Health Record (EHR) Analysis
• SeqSleepNet: End-to-End Hierarchical Recurrent Neural Network for Sequence-to-Sequence Automatic Sleep Staging
• Learning to Diagnose with LSTM Recurrent Neural Networks
• Semi-supervised Sequence Learning
• Hierarchical Multiscale Recurrent Neural Networks
• Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
• DeepCare: A Deep Dynamic Memory Model for Predictive Medicine
• Multi-Label Classification of Patient Notes a Case Study on ICD Code Assignment
• Deep EHR: Chronic Disease Prediction Using Medical Notes
• Grounded Recurrent Neural Networks

Cited by in corpus (6)

• Deep Representation Learning of Patient Data from Electronic Health Records (EHR): A Systematic Review
• A Deep Learning Pipeline for Patient Diagnosis Prediction Using Electronic Health Records
• Generalized and Transferable Patient Language Representation for Phenotyping with Limited Data
• Explainable Health Risk Predictor with Transformer-based Medicare Claim Encoder
• MUFASA: Multimodal Fusion Architecture Search for Electronic Health Records
• Does Robustness Improve Fairness? Approaching Fairness with Word Substitution Robustness Methods for Text Classification