Hierarchical Label-wise Attention Transformer Model for Explainable ICD Coding

Abstract

International Classification of Diseases (ICD) coding plays an important role in systematically classifying morbidity and mortality data. In this study, we propose a hierarchical label-wise attention Transformer model (HiLAT) for the explainable prediction of ICD codes from clinical documents. HiLAT firstly fine-tunes a pretrained Transformer model to represent the tokens of clinical documents. We subsequently employ a two-level hierarchical label-wise attention mechanism that creates label-specific document representations. These representations are in turn used by a feed-forward neural network to predict whether a specific ICD code is assigned to the input clinical document of interest. We evaluate HiLAT using hospital discharge summaries and their corresponding ICD-9 codes from the MIMIC-III database. To investigate the performance of different types of Transformer models, we develop ClinicalplusXLNet, which conducts continual pretraining from XLNet-Base using all the MIMIC-III clinical notes. The experiment results show that the F1 scores of the HiLAT+ClinicalplusXLNet outperform the previous state-of-the-art models for the top-50 most frequent ICD-9 codes from MIMIC-III. Visualisations of attention weights present a potential explainability tool for checking the face validity of ICD code predictions.

References in corpus (13)

• Efficient Estimation of Word Representations in Vector Space
• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• Longformer: The Long-Document Transformer
• Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
• European Union regulations on algorithmic decision-making and a "right to explanation"
• SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
• Publicly Available Clinical BERT Embeddings
• ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
• Attention is not Explanation
• A Label Attention Model for ICD Coding from Clinical Text
• Dilated Convolutional Attention Network for Medical Code Assignment from Clinical Text
• Medical code prediction with multi-view convolution and description-regularized label-dependent attention
• A Meta-embedding-based Ensemble Approach for ICD Coding Prediction

Cited by in corpus (2)

• Automated ICD Coding using Extreme Multi-label Long Text Transformer-based Models
• A Unified Review of Deep Learning for Automated Medical Coding