A scoping review on multimodal deep learning in biomedical images and texts
arXiv:2307.07362 · doi:10.1016/j.jbi.2023.104482
Abstract
Computer-assisted diagnostic and prognostic systems of the future should be capable of simultaneously processing multimodal data. Multimodal deep learning (MDL), which involves the integration of multiple sources of data, such as images and text, has the potential to revolutionize the analysis and interpretation of biomedical data. However, it only caught researchers' attention recently. To this end, there is a critical need to conduct a systematic review on this topic, identify the limitations of current work, and explore future directions. In this scoping review, we aim to provide a comprehensive overview of the current state of the field and identify key concepts, types of studies, and research gaps with a focus on biomedical images and texts joint learning, mainly because these two were the most commonly available data types in MDL research. This study reviewed the current uses of multimodal deep learning on five tasks: (1) Report generation, (2) Visual question answering, (3) Cross-modal retrieval, (4) Computer-aided diagnosis, and (5) Semantic segmentation. Our results highlight the diverse applications and potential of MDL and suggest directions for future research in the field. We hope our review will facilitate the collaboration of natural language processing (NLP) and medical imaging communities and support the next generation of decision-making and computer-assisted diagnostic system development.
arXiv commentThis paper has been accepted by the Journal of Biomedical Informatics
References in corpus (10)
- A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAI
- Explainable artificial intelligence (XAI) in deep learning-based medical image analysis
- Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
- Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing
- Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training
- Deep learning for temporal data representation in electronic health records: A systematic review of challenges and methodologies
- Medical-VLBERT: Medical Visual Language BERT for COVID-19 CT Report Generation With Alternate Learning
- Multi-Granularity Cross-modal Alignment for Generalized Medical Visual Representation Learning
- RepsNet: Combining Vision with Language for Automated Medical Reports
- MMGL: Multi-Scale Multi-View Global-Local Contrastive learning for Semi-supervised Cardiac Image Segmentation