Weakly-Supervised Convolutional Neural Networks for Multimodal Image Registration

Abstract

One of the fundamental challenges in supervised learning for multimodal image registration is the lack of ground-truth for voxel-level spatial correspondence. This work describes a method to infer voxel-level transformation from higher-level correspondence information contained in anatomical labels. We argue that such labels are more reliable and practical to obtain for reference sets of image pairs than voxel-level correspondence. Typical anatomical labels of interest may include solid organs, vessels, ducts, structure boundaries and other subject-specific ad hoc landmarks. The proposed end-to-end convolutional neural network approach aims to predict displacement fields to align multiple labelled corresponding structures for individual image pairs during the training, while only unlabelled image pairs are used as the network input for inference. We highlight the versatility of the proposed strategy, for training, utilising diverse types of anatomical labels, which need not to be identifiable over all training image pairs. At inference, the resulting 3D deformable image registration algorithm runs in real-time and is fully-automated without requiring any anatomical labels or initialisation. Several network architecture variants are compared for registering T2-weighted magnetic resonance images and 3D transrectal ultrasound images from prostate cancer patients. A median target registration error of 3.6 mm on landmark centroids and a median Dice of 0.87 on prostate glands are achieved from cross-validation experiments, in which 108 pairs of multimodal images from 76 patients were tested with high-quality anatomical labels.

Accepted manuscript in Medical Image Analysis

References in corpus (9)

• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations
• Deeply-Supervised Nets
• Interactive Medical Image Segmentation using Deep Learning with Image-specific Fine-tuning
• FlowNet: Learning Optical Flow with Convolutional Networks
• NiftyNet: a deep-learning platform for medical imaging
• End-to-End Unsupervised Deformable Image Registration with a Convolutional Neural Network
• ToolNet: Holistically-Nested Real-Time Segmentation of Robotic Surgical Tools
• Label-driven weakly-supervised learning for multimodal deformable image registration

Cited by in corpus (57)

• VoxelMorph: A Learning Framework for Deformable Medical Image Registration
• Recent advances and clinical applications of deep learning in medical image analysis
• A Deep Learning Framework for Unsupervised Affine and Deformable Image Registration
• Deep Learning in Medical Image Registration: A Review
• Deep Learning in Medical Image Registration: A Survey
• Unsupervised Learning of Probabilistic Diffeomorphic Registration for Images and Surfaces
• LasHeR: A Large-scale High-diversity Benchmark for RGBT Tracking
• Learning a Probabilistic Model for Diffeomorphic Registration
• SynthMorph: learning contrast-invariant registration without acquired images
• Medical Image Registration Using Deep Neural Networks: A Comprehensive Review
• CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer Assisted Interventions
• Learning Deformable Registration of Medical Images with Anatomical Constraints
• Automatic Registration between Cone-Beam CT and Scanned Surface via Deep-Pose Regression Neural Networks and Clustered Similarities
• DeepReg: a deep learning toolkit for medical image registration
• Learning the Effect of Registration Hyperparameters with HyperMorph
• $\mathcal{X}$-Metric: An N-Dimensional Information-Theoretic Framework for Groupwise Registration and Deep Combined Computing
• AutoFuse: Automatic Fusion Networks for Deformable Medical Image Registration
• Transformation Consistent Self-ensembling Model for Semi-supervised Medical Image Segmentation
• Enhancing Medical Image Registration via Appearance Adjustment Networks
• Residual Aligner Network
• FDRN: A Fast Deformable Registration Network for Medical Images
• An Unsupervised Approach to Ultrasound Elastography with End-to-end Strain Regularisation
• Cross-Modality Image Registration using a Training-Time Privileged Third Modality
• Multi-Modal Learning for AU Detection Based on Multi-Head Fused Transformers
• Conditional Segmentation in Lieu of Image Registration
• Recursive Deformable Image Registration Network with Mutual Attention
• A Two-Step Deep Learning Method for 3DCT-2DUS Kidney Registration During Breathing
• Image-and-Spatial Transformer Networks for Structure-Guided Image Registration
• Few Labeled Atlases are Necessary for Deep-Learning-Based Segmentation
• Cross-modal Attention for MRI and Ultrasound Volume Registration
• Learning deep abdominal CT registration through adaptive loss weighting and synthetic data generation
• Longitudinal Image Registration with Temporal-order and Subject-specificity Discrimination
• Unsupervised Deformable Medical Image Registration via Pyramidal Residual Deformation Fields Estimation
• Tackling the Problem of Large Deformations in Deep Learning Based Medical Image Registration Using Displacement Embeddings
• Morphological Change Forecasting for Prostate Glands using Feature-based Registration and Kernel Density Extrapolation
• Adversarial Uni- and Multi-modal Stream Networks for Multimodal Image Registration
• Deep Complementary Joint Model for Complex Scene Registration and Few-shot Segmentation on Medical Images
• Nonrigid Reconstruction of Freehand Ultrasound without a Tracker
• F3RNet: Full-Resolution Residual Registration Network for Deformable Image Registration
• Unimodal Cyclic Regularization for Training Multimodal Image Registration Networks
• Deformable Medical Image Registration Using a Randomly-Initialized CNN as Regularization Prior
• A low-rank representation for unsupervised registration of medical images
• One-shot Joint Extraction, Registration and Segmentation of Neuroimaging Data
• KLDivNet: An unsupervised neural network for multi-modality image registration
• Impact of PCA-based preprocessing and different CNN structures on deformable registration of sonograms
• JSSR: A Joint Synthesis, Segmentation, and Registration System for 3D Multi-Modal Image Alignment of Large-scale Pathological CT Scans
• Atlas-Based Segmentation of Intracochlear Anatomy in Metal Artifact Affected CT Images of the Ear with Co-trained Deep Neural Networks
• Unsupervised Deformable Registration for Multi-Modal Images via Disentangled Representations
• Bayesian Unsupervised Disentanglement of Anatomy and Geometry for Deep Groupwise Image Registration
• Artificial Intelligence in Quantitative Ultrasound Imaging: A Review
• MICS : Multi-steps, Inverse Consistency and Symmetric deep learning registration network
• MvMM-RegNet: A new image registration framework based on multivariate mixture model and neural network estimation
• Anatomy-guided Multimodal Registration by Learning Segmentation without Ground Truth: Application to Intraprocedural CBCT/MR Liver Segmentation and Registration
• Real-time multimodal image registration with partial intraoperative point-set data
• Unsupervised Multimodal Image Registration with Adaptative Gradient Guidance
• Data-driven registration and modeling of brain deformation for image-guided neurosurgery
• End-to-end Ultrasound Frame to Volume Registration