Does the Magic of BERT Apply to Medical Code Assignment? A Quantitative Study

Abstract

Unsupervised pretraining is an integral part of many natural language processing systems, and transfer learning with language models has achieved remarkable results in many downstream tasks. In the clinical application of medical code assignment, diagnosis and procedure codes are inferred from lengthy clinical notes such as hospital discharge summaries. However, it is not clear if pretrained models are useful for medical code prediction without further architecture engineering. This paper conducts a comprehensive quantitative analysis of various contextualized language models' performance, pretrained in different domains, for medical code assignment from clinical notes. We propose a hierarchical fine-tuning architecture to capture interactions between distant words and adopt label-wise attention to exploit label information. Contrary to current trends, we demonstrate that a carefully trained classical CNN outperforms attention-based models on a MIMIC-III subset with frequent codes. Our empirical findings suggest directions for improving the medical code assignment application.

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Distributed Representations of Words and Phrases and their Compositionality
• Decoupled Weight Decay Regularization
• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• Longformer: The Long-Document Transformer
• Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
• Pre-trained Models for Natural Language Processing: A Survey
• Convolutional Neural Networks for Sentence Classification
• Linformer: Self-Attention with Linear Complexity
• ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
• Big Bird: Transformers for Longer Sequences
• Towards Automated ICD Coding Using Deep Learning
• Phenotyping of Clinical Notes with Improved Document Classification Models Using Contextualized Neural Language Models

Cited by in corpus (6)

• A Unified Review of Deep Learning for Automated Medical Coding
• Multitask Balanced and Recalibrated Network for Medical Code Prediction
• Multitask Recalibrated Aggregation Network for Medical Code Prediction
• Efficacy of BERT embeddings on predicting disaster from Twitter data
• Description-based Label Attention Classifier for Explainable ICD-9 Classification
• CoPHE: A Count-Preserving Hierarchical Evaluation Metric in Large-Scale Multi-Label Text Classification