A deep learning framework for quality assessment and restoration in video endoscopy

Abstract

Endoscopy is a routine imaging technique used for both diagnosis and minimally invasive surgical treatment. Artifacts such as motion blur, bubbles, specular reflections, floating objects and pixel saturation impede the visual interpretation and the automated analysis of endoscopy videos. Given the widespread use of endoscopy in different clinical applications, we contend that the robust and reliable identification of such artifacts and the automated restoration of corrupted video frames is a fundamental medical imaging problem. Existing state-of-the-art methods only deal with the detection and restoration of selected artifacts. However, typically endoscopy videos contain numerous artifacts which motivates to establish a comprehensive solution. We propose a fully automatic framework that can: 1) detect and classify six different primary artifacts, 2) provide a quality score for each frame and 3) restore mildly corrupted frames. To detect different artifacts our framework exploits fast multi-scale, single stage convolutional neural network detector. We introduce a quality metric to assess frame quality and predict image restoration success. Generative adversarial networks with carefully chosen regularization are finally used to restore corrupted frames. Our detector yields the highest mean average precision (mAP at 5% threshold) of 49.0 and the lowest computational time of 88 ms allowing for accurate real-time processing. Our restoration models for blind deblurring, saturation correction and inpainting demonstrate significant improvements over previous methods. On a set of 10 test videos we show that our approach preserves an average of 68.7% which is 25% more frames than that retained from the raw videos.

14 pages

References in corpus (11)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Conditional Generative Adversarial Nets
• YOLOv3: An Incremental Improvement
• Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
• Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition
• Progressive Growing of GANs for Improved Quality, Stability, and Variation
• A review: Deep learning for medical image segmentation using multi-modality fusion
• Generative Image Inpainting with Contextual Attention
• Abnormal Colon Polyp Image Synthesis Using Conditional Adversarial Networks for Improved Detection Performance
• Polyp Detection and Segmentation from Video Capsule Endoscopy: A Review
• Improved Training of Generative Adversarial Networks Using Representative Features

Cited by in corpus (12)

• Real-Time Polyp Detection, Localization and Segmentation in Colonoscopy Using Deep Learning
• Deep learning for detection and segmentation of artefact and disease instances in gastrointestinal endoscopy
• Deep learning in biomedical optics
• Artifact- and content-specific quality assessment for MRI with image rulers
• SimCol3D -- 3D Reconstruction during Colonoscopy Challenge
• Understanding the effects of artifacts on automated polyp detection and incorporating that knowledge via learning without forgetting
• How can we learn (more) from challenges? A statistical approach to driving future algorithm development
• Specular reflections removal in colposcopic images based on neural networks: Supervised training with no ground truth previous knowledge
• Multi-Plateau Ensemble for Endoscopic Artefact Segmentation and Detection
• Polyp-artifact relationship analysis using graph inductive learned representations
• Colorectal Polyp Detection in Real-world Scenario: Design and Experiment Study
• Panoptic Segmentation: A Review