A review: Deep learning for medical image segmentation using multi-modality fusion

Abstract

Multi-modality is widely used in medical imaging, because it can provide multiinformation about a target (tumor, organ or tissue). Segmentation using multimodality consists of fusing multi-information to improve the segmentation. Recently, deep learning-based approaches have presented the state-of-the-art performance in image classification, segmentation, object detection and tracking tasks. Due to their self-learning and generalization ability over large amounts of data, deep learning recently has also gained great interest in multi-modal medical image segmentation. In this paper, we give an overview of deep learning-based approaches for multi-modal medical image segmentation task. Firstly, we introduce the general principle of deep learning and multi-modal medical image segmentation. Secondly, we present different deep learning network architectures, then analyze their fusion strategies and compare their results. The earlier fusion is commonly used, since it's simple and it focuses on the subsequent segmentation network architecture. However, the later fusion gives more attention on fusion strategy to learn the complex relationship between different modalities. In general, compared to the earlier fusion, the later fusion can give more accurate result if the fusion method is effective enough. We also discuss some common problems in medical image segmentation. Finally, we summarize and provide some perspectives on the future research.

26 pages, 8 figures

References in corpus (18)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• SMOTE: Synthetic Minority Over-sampling Technique
• A Survey on Deep Learning in Medical Image Analysis
• Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentation
• Brain Tumor Segmentation with Deep Neural Networks
• Fully Convolutional Networks for Semantic Segmentation
• Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations
• The Effectiveness of Data Augmentation in Image Classification using Deep Learning
• Focal Loss for Dense Object Detection
• Domain-adversarial neural networks to address the appearance variability of histopathology images
• A deep learning model integrating FCNNs and CRFs for brain tumor segmentation
• Automatic Brain Tumor Segmentation using Cascaded Anisotropic Convolutional Neural Networks
• DeepOrgan: Multi-level Deep Convolutional Networks for Automated Pancreas Segmentation
• Ensembles of Multiple Models and Architectures for Robust Brain Tumour Segmentation
• 3D Densely Convolutional Networks for Volumetric Segmentation
• Multi Modal Convolutional Neural Networks for Brain Tumor Segmentation
• MRI Tumor Segmentation with Densely Connected 3D CNN
• Autofocus Layer for Semantic Segmentation

Cited by in corpus (41)

• Medical Image Segmentation Using Deep Learning: A Survey
• Modality specific U-Net variants for biomedical image segmentation: A survey
• Latent Correlation Representation Learning for Brain Tumor Segmentation with Missing MRI Modalities
• A deep learning framework for quality assessment and restoration in video endoscopy
• A Systematic Review of Intermediate Fusion in Multimodal Deep Learning for Biomedical Applications
• Feature-enhanced Generation and Multi-modality Fusion based Deep Neural Network for Brain Tumor Segmentation with Missing MR Modalities
• PathoNet: Deep learning assisted evaluation of Ki-67 and tumor infiltrating lymphocytes (TILs) as prognostic factors in breast cancer; A large dataset and baseline
• MyoPS-Net: Myocardial Pathology Segmentation with Flexible Combination of Multi-Sequence CMR Images
• Application of belief functions to medical image segmentation: A review
• Brain tumor segmentation with missing modalities via latent multi-source correlation representation
• Enhancing Network Initialization for Medical AI Models Using Large-Scale, Unlabeled Natural Images
• Prediction of brain tumor recurrence location based on multi-modal fusion and nonlinear correlation learning
• MouseGAN++: Unsupervised Disentanglement and Contrastive Representation for Multiple MRI Modalities Synthesis and Structural Segmentation of Mouse Brain
• MAG-Net: Multi-task attention guided network for brain tumor segmentation and classification
• Deep Negative Volume Segmentation
• Deep Multimodal Fusion of Data with Heterogeneous Dimensionality via Projective Networks
• Multi-source adversarial transfer learning for ultrasound image segmentation with limited similarity
• Lesion-Aware Cross-Phase Attention Network for Renal Tumor Subtype Classification on Multi-Phase CT Scans
• Resource-Frugal Classification and Analysis of Pathology Slides Using Image Entropy
• TransMed: Transformers Advance Multi-modal Medical Image Classification
• Chest X-ray Image Phase Features for Improved Diagnosis of COVID-19 Using Convolutional Neural Network
• ACN: Adversarial Co-training Network for Brain Tumor Segmentation with Missing Modalities
• Multi-modal AsynDGAN: Learn From Distributed Medical Image Data without Sharing Private Information
• A Robotic 3D Perception System for Operating Room Environment Awareness
• Robust Pancreatic Ductal Adenocarcinoma Segmentation with Multi-Institutional Multi-Phase Partially-Annotated CT Scans
• MMFNet: A Multi-modality MRI Fusion Network for Segmentation of Nasopharyngeal Carcinoma
• Noisy Labels are Treasure: Mean-Teacher-Assisted Confident Learning for Hepatic Vessel Segmentation
• A Simple and Robust Framework for Cross-Modality Medical Image Segmentation applied to Vision Transformers
• Towards Label-Free 3D Segmentation of Optical Coherence Tomography Images of the Optic Nerve Head Using Deep Learning
• Brain Tumor Segmentation Network Using Attention-based Fusion and Spatial Relationship Constraint
• Automatic tracing of mandibular canal pathways using deep learning
• Uncertainty quantification in medical image segmentation with normalizing flows
• Ensemble Methods for Multi-Organ Segmentation in CT Series
• Multi-Modality Pathology Segmentation Framework: Application to Cardiac Magnetic Resonance Images
• Comparing Deep Learning strategies for paired but unregistered multimodal segmentation of the liver in T1 and T2-weighted MRI
• A Tri-attention Fusion Guided Multi-modal Segmentation Network
• 3D Medical Multi-modal Segmentation Network Guided by Multi-source Correlation Constraint
• Feasibility Assessment of Multitasking in MRI Neuroimaging Analysis: Tissue Segmentation, Cross-Modality Conversion and Bias correction
• A new smart-cropping pipeline for prostate segmentation using deep learning networks
• LORCK: Learnable Object-Resembling Convolution Kernels
• Toward High-Throughput Artificial Intelligence-Based Segmentation in Oncological PET Imaging