Mutually improved endoscopic image synthesis and landmark detection in unpaired image-to-image translation

Abstract

The CycleGAN framework allows for unsupervised image-to-image translation of unpaired data. In a scenario of surgical training on a physical surgical simulator, this method can be used to transform endoscopic images of phantoms into images which more closely resemble the intra-operative appearance of the same surgical target structure. This can be viewed as a novel augmented reality approach, which we coined Hyperrealism in previous work. In this use case, it is of paramount importance to display objects like needles, sutures or instruments consistent in both domains while altering the style to a more tissue-like appearance. Segmentation of these objects would allow for a direct transfer, however, contouring of these, partly tiny and thin foreground objects is cumbersome and perhaps inaccurate. Instead, we propose to use landmark detection on the points when sutures pass into the tissue. This objective is directly incorporated into a CycleGAN framework by treating the performance of pre-trained detector models as an additional optimization goal. We show that a task defined on these sparse landmark labels improves consistency of synthesis by the generator network in both domains. Comparing a baseline CycleGAN architecture to our proposed extension (DetCycleGAN), mean precision (PPV) improved by +61.32, mean sensitivity (TPR) by +37.91, and mean F1 score by +0.4743. Furthermore, it could be shown that by dataset fusion, generated intra-operative images can be leveraged as additional training data for the detection network itself. The data is released within the scope of the AdaptOR MICCAI Challenge 2021 at https://adaptor2021.github.io/, and code at https://github.com/Cardio-AI/detcyclegan_pytorch.

arXiv commentAccepted for IEEE JBHI 2021, 13 pages, 8 figures, 4 tables

References in corpus (15)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Albumentations: fast and flexible image augmentations
• Generative Adversarial Network in Medical Imaging: A Review
• Toward Multimodal Image-to-Image Translation
• Curriculum Domain Adaptation for Semantic Segmentation of Urban Scenes
• Integrating Spatial Configuration into Heatmap Regression Based CNNs for Landmark Localization
• Unsupervised Learning of Disentangled Representations from Video
• Cross-Domain Car Detection Using Unsupervised Image-to-Image Translation: From Day to Night
• Structured Landmark Detection via Topology-Adapting Deep Graph Learning
• Flexible and Comprehensive Patient-Specific Mitral Valve Silicone Models with Chordae Tendinae Made From 3D-Printable Molds
• Unsupervised Domain Adaptation for Object Detection via Cross-Domain Semi-Supervised Learning
• Improving Surgical Training Phantoms by Hyperrealism: Deep Unpaired Image-to-Image Translation from Real Surgeries
• Landmark Assisted CycleGAN for Cartoon Face Generation
• Image-to-Image Translation: Methods and Applications
• Point detection through multi-instance deep heatmap regression for sutures in endoscopy

Cited by in corpus (4)

• Point detection through multi-instance deep heatmap regression for sutures in endoscopy
• This Intestine Does Not Exist: Multiscale Residual Variational Autoencoder for Realistic Wireless Capsule Endoscopy Image Generation
• Improved Heatmap-based Landmark Detection
• Multiscale Vector-Quantized Variational Autoencoder for Endoscopic Image Synthesis