A Simple and Robust Framework for Cross-Modality Medical Image Segmentation applied to Vision Transformers

Abstract

When it comes to clinical images, automatic segmentation has a wide variety of applications and a considerable diversity of input domains, such as different types of Magnetic Resonance Images (MRIs) and Computerized Tomography (CT) scans. This heterogeneity is a challenge for cross-modality algorithms that should equally perform independently of the input image type fed to them. Often, segmentation models are trained using a single modality, preventing generalization to other types of input data without resorting to transfer learning techniques. Furthermore, the multi-modal or cross-modality architectures proposed in the literature frequently require registered images, which are not easy to collect in clinical environments, or need additional processing steps, such as synthetic image generation. In this work, we propose a simple framework to achieve fair image segmentation of multiple modalities using a single conditional model that adapts its normalization layers based on the input type, trained with non-registered interleaved mixed data. We show that our framework outperforms other cross-modality segmentation methods, when applied to the same 3D UNet baseline model, on the Multi-Modality Whole Heart Segmentation Challenge. Furthermore, we define the Conditional Vision Transformer (C-ViT) encoder, based on the proposed cross-modality framework, and we show that it brings significant improvements to the resulting segmentation, up to 6.87\% of Dice accuracy, with respect to its baseline reference. The code to reproduce our experiments and the trained model weights are available at https://github.com/matteo-bastico/MI-Seg.

arXiv commentThis paper has been accepted in International Conference on Computer Vision Workshops (ICCVW) 2023

References in corpus (17)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Distilling the Knowledge in a Neural Network
• Decoupled Weight Decay Regularization
• Generative Adversarial Networks
• TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
• Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?
• Instance Normalization: The Missing Ingredient for Fast Stylization
• U-Net and its variants for medical image segmentation: theory and applications
• A survey of loss functions for semantic segmentation
• A Learned Representation For Artistic Style
• A review: Deep learning for medical image segmentation using multi-modality fusion
• ResViT: Residual vision transformers for multi-modal medical image synthesis
• MONAI: An open-source framework for deep learning in healthcare
• How Do Vision Transformers Work?
• Medical Image Segmentation Based on Multi-Modal Convolutional Neural Network: Study on Image Fusion Schemes
• Scalable multimodal convolutional networks for brain tumour segmentation