Disentangled Representation Learning in Cardiac Image Analysis

Abstract

Typically, a medical image offers spatial information on the anatomy (and pathology) modulated by imaging specific characteristics. Many imaging modalities including Magnetic Resonance Imaging (MRI) and Computed Tomography (CT) can be interpreted in this way. We can venture further and consider that a medical image naturally factors into some spatial factors depicting anatomy and factors that denote the imaging characteristics. Here, we explicitly learn this decomposed (disentangled) representation of imaging data, focusing in particular on cardiac images. We propose Spatial Decomposition Network (SDNet), which factorises 2D medical images into spatial anatomical factors and non-spatial modality factors. We demonstrate that this high-level representation is ideally suited for several medical image analysis tasks, such as semi-supervised segmentation, multi-task segmentation and regression, and image-to-image synthesis. Specifically, we show that our model can match the performance of fully supervised segmentation models, using only a fraction of the labelled images. Critically, we show that our factorised representation also benefits from supervision obtained either when we use auxiliary tasks to train the model in a multi-task setting (e.g. regressing to known cardiac indices), or when aggregating multimodal data from different sources (e.g. pooling together MRI and CT data). To explore the properties of the learned factorisation, we perform latent-space arithmetic and show that we can synthesise CT from MR and vice versa, by swapping the modality factors. We also demonstrate that the factor holding image specific information can be used to predict the input modality with high accuracy. Code will be made available at https://github.com/agis85/anatomy_modality_decomposition.

References in corpus (16)

• Adam: A Method for Stochastic Optimization
• Auto-Encoding Variational Bayes
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
• Toward Multimodal Image-to-Image Translation
• Anatomically Constrained Neural Networks (ACNN): Application to Cardiac Image Enhancement and Segmentation
• Disentangling by Factorising
• Augmented CycleGAN: Learning Many-to-Many Mappings from Unpaired Data
• Multimodal Unsupervised Image-to-Image Translation
• Understanding disentangling in $\beta$-VAE
• Disentangling factors of variation in deep representations using adversarial training
• Scalable multimodal convolutional networks for brain tumour segmentation
• Challenges in Disentangling Independent Factors of Variation
• Disentangling Factors of Variation by Mixing Them
• Deep Learning Based Instance Segmentation in 3D Biomedical Images Using Weak Annotation

Cited by in corpus (37)

• A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises
• Deep learning for cardiac image segmentation: A review
• Learning to Segment from Scribbles using Multi-scale Adversarial Attention Gates
• Automated Detection and Forecasting of COVID-19 using Deep Learning Techniques: A Review
• Learning Disentangled Representations in the Imaging Domain
• Weakly-Supervised Disentanglement Without Compromises
• Is attention all you need in medical image analysis? A review
• Learning to synthesise the ageing brain without longitudinal data
• Disentangle, align and fuse for multimodal and semi-supervised image segmentation
• Pseudo-healthy synthesis with pathology disentanglement and adversarial learning
• MT-UDA: Towards Unsupervised Cross-modality Medical Image Segmentation with Limited Source Labels
• Segmentation-Renormalized Deep Feature Modulation for Unpaired Image Harmonization
• Conditional Generation of Medical Images via Disentangled Adversarial Inference
• MouseGAN++: Unsupervised Disentanglement and Contrastive Representation for Multiple MRI Modalities Synthesis and Structural Segmentation of Mouse Brain
• Unified Brain MR-Ultrasound Synthesis using Multi-Modal Hierarchical Representations
• Spatio-temporal Multi-task Learning for Cardiac MRI Left Ventricle Quantification
• SE(3)-Equivariant and Noise-Invariant 3D Rigid Motion Tracking in Brain MRI
• Learning Cross-domain Generalizable Features by Representation Disentanglement
• Contrastive Semi-Supervised Learning for 2D Medical Image Segmentation
• GLOWin: A Flow-based Invertible Generative Framework for Learning Disentangled Feature Representations in Medical Images
• Temporal Consistency Objectives Regularize the Learning of Disentangled Representations
• FedDis: Disentangled Federated Learning for Unsupervised Brain Pathology Segmentation
• 4D Semantic Cardiac Magnetic Resonance Image Synthesis on XCAT Anatomical Model
• Recent Advances in Fibrosis and Scar Segmentation from Cardiac MRI: A State-of-the-Art Review and Future Perspectives
• Re-using Adversarial Mask Discriminators for Test-time Training under Distribution Shifts
• Measuring the Biases and Effectiveness of Content-Style Disentanglement
• GAN-based disentanglement learning for chest X-ray rib suppression
• Algorithm Fairness in AI for Medicine and Healthcare
• Decomposing Normal and Abnormal Features of Medical Images into Discrete Latent Codes for Content-Based Image Retrieval
• Echocardiography Segmentation with Enforced Temporal Consistency
• Semi-supervised Meta-learning with Disentanglement for Domain-generalised Medical Image Segmentation
• Unified Cross-Modal Medical Image Synthesis with Hierarchical Mixture of Product-of-Experts
• Unsupervised Anomaly Detection in Brain MRI via Disentangled Anatomy Learning
• Disentangled Representations for Domain-generalized Cardiac Segmentation
• SD-LayerNet: Semi-supervised retinal layer segmentation in OCT using disentangled representation with anatomical priors
• Semi-supervised Pathology Segmentation with Disentangled Representations
• Mutual Information-based Disentangled Neural Networks for Classifying Unseen Categories in Different Domains: Application to Fetal Ultrasound Imaging