Unpaired Multi-modal Segmentation via Knowledge Distillation

Abstract

Multi-modal learning is typically performed with network architectures containing modality-specific layers and shared layers, utilizing co-registered images of different modalities. We propose a novel learning scheme for unpaired cross-modality image segmentation, with a highly compact architecture achieving superior segmentation accuracy. In our method, we heavily reuse network parameters, by sharing all convolutional kernels across CT and MRI, and only employ modality-specific internal normalization layers which compute respective statistics. To effectively train such a highly compact model, we introduce a novel loss term inspired by knowledge distillation, by explicitly constraining the KL-divergence of our derived prediction distributions between modalities. We have extensively validated our approach on two multi-class segmentation problems: i) cardiac structure segmentation, and ii) abdominal organ segmentation. Different network settings, i.e., 2D dilated network and 3D U-net, are utilized to investigate our method's general efficacy. Experimental results on both tasks demonstrate that our novel multi-modal learning scheme consistently outperforms single-modal training and previous multi-modal approaches.

IEEE TMI, code available

References in corpus (7)

• Distilling the Knowledge in a Neural Network
• Efficient Multi-Scale 3D CNN with Fully Connected CRF for Accurate Brain Lesion Segmentation
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Deep Learning for Multi-Task Medical Image Segmentation in Multiple Modalities
• Multi-source Transfer Learning with Convolutional Neural Networks for Lung Pattern Analysis
• Universal representations:The missing link between faces, text, planktons, and cat breeds
• Scalable multimodal convolutional networks for brain tumour segmentation

Cited by in corpus (5)

• Knowledge Distillation: A Survey
• Contrastive Cross-site Learning with Redesigned Net for COVID-19 CT Classification
• Source-Free Domain Adaptation for Semantic Segmentation
• Unsupervised learning of multimodal image registration using domain adaptation with projected Earth Move's discrepancies
• EvDistill: Asynchronous Events to End-task Learning via Bidirectional Reconstruction-guided Cross-modal Knowledge Distillation