Enhancing chest X-ray datasets with privacy-preserving large language models and multi-type annotations: a data-driven approach for improved classification

Abstract

In chest X-ray (CXR) image analysis, rule-based systems are usually employed to extract labels from reports for dataset releases. However, there is still room for improvement in label quality. These labelers typically output only presence labels, sometimes with binary uncertainty indicators, which limits their usefulness. Supervised deep learning models have also been developed for report labeling but lack adaptability, similar to rule-based systems. In this work, we present MAPLEZ (Medical report Annotations with Privacy-preserving Large language model using Expeditious Zero shot answers), a novel approach leveraging a locally executable Large Language Model (LLM) to extract and enhance findings labels on CXR reports. MAPLEZ extracts not only binary labels indicating the presence or absence of a finding but also the location, severity, and radiologists' uncertainty about the finding. Over eight abnormalities from five test sets, we show that our method can extract these annotations with an increase of 3.6 percentage points (pp) in macro F1 score for categorical presence annotations and more than 20 pp increase in F1 score for the location annotations over competing labelers. Additionally, using the combination of improved annotations and multi-type annotations in classification supervision, we demonstrate substantial advancements in model quality, with an increase of 1.1 pp in AUROC over models trained with annotations from the best alternative approach. We share code and annotations.

arXiv commentCode and data: https://github.com/rsummers11/CADLab/tree/master/MAPLEZ_LLM_report_labeler/

References in corpus (14)

• Llama 2: Open Foundation and Fine-Tuned Chat Models
• BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining
• PadChest: A large chest x-ray image dataset with multi-label annotated reports
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Galactica: A Large Language Model for Science
• MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs
• NegBio: a high-performance tool for negation and uncertainty detection in radiology reports
• The Flan Collection: Designing Data and Methods for Effective Instruction Tuning
• Machine-Learning-Based Multiple Abnormality Prediction with Large-Scale Chest Computed Tomography Volumes
• Orca: Progressive Learning from Complex Explanation Traces of GPT-4
• REFLACX, a dataset of reports and eye-tracking data for localization of abnormalities in chest x-rays
• Interleaved Text/Image Deep Mining on a Large-Scale Radiology Database for Automated Image Interpretation
• CheX-GPT: Harnessing Large Language Models for Enhanced Chest X-ray Report Labeling
• Expert Uncertainty and Severity Aware Chest X-Ray Classification by Multi-Relationship Graph Learning

Cited by in corpus (1)

• From large language models to multimodal AI: A scoping review on the potential of generative AI in medicine