Latent Correlation Representation Learning for Brain Tumor Segmentation with Missing MRI Modalities

Abstract

Magnetic Resonance Imaging (MRI) is a widely used imaging technique to assess brain tumor. Accurately segmenting brain tumor from MR images is the key to clinical diagnostics and treatment planning. In addition, multi-modal MR images can provide complementary information for accurate brain tumor segmentation. However, it's common to miss some imaging modalities in clinical practice. In this paper, we present a novel brain tumor segmentation algorithm with missing modalities. Since it exists a strong correlation between multi-modalities, a correlation model is proposed to specially represent the latent multi-source correlation. Thanks to the obtained correlation representation, the segmentation becomes more robust in the case of missing modality. First, the individual representation produced by each encoder is used to estimate the modality independent parameter. Then, the correlation model transforms all the individual representations to the latent multi-source correlation representations. Finally, the correlation representations across modalities are fused via attention mechanism into a shared representation to emphasize the most important features for segmentation. We evaluate our model on BraTS 2018 and BraTS 2019 dataset, it outperforms the current state-of-the-art methods and produces robust results when one or more modalities are missing.

12 pages, 10 figures, accepted by IEEE Transactions on Image Processing (8 April 2021). arXiv admin note: text overlap with arXiv:2003.08870, arXiv:2102.03111

References in corpus (10)

• Attention U-Net: Learning Where to Look for the Pancreas
• Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentation
• Brain Tumor Segmentation with Deep Neural Networks
• A review: Deep learning for medical image segmentation using multi-modality fusion
• Automatic Brain Tumor Segmentation using Cascaded Anisotropic Convolutional Neural Networks
• Cross-Modality Deep Feature Learning for Brain Tumor Segmentation
• Pyramid Attention Network for Semantic Segmentation
• One-pass Multi-task Networks with Cross-task Guided Attention for Brain Tumor Segmentation
• Brain Tumor Synthetic Segmentation in 3D Multimodal MRI Scans
• A unified representation network for segmentation with missing modalities

Cited by in corpus (5)

• Feature-enhanced Generation and Multi-modality Fusion based Deep Neural Network for Brain Tumor Segmentation with Missing MR Modalities
• Unified Multi-Modal Image Synthesis for Missing Modality Imputation
• A Quantitative Comparison between Shannon and Tsallis Havrda Charvat Entropies Applied to Cancer Outcome Prediction
• Brain Tumor Detection Based on a Novel and High-Quality Prediction of the Tumor Pixel Distributions
• Federated Modality-specific Encoders and Partially Personalized Fusion Decoder for Multimodal Brain Tumor Segmentation