Robust deep learning-based semantic organ segmentation in hyperspectral images

Abstract

Semantic image segmentation is an important prerequisite for context-awareness and autonomous robotics in surgery. The state of the art has focused on conventional RGB video data acquired during minimally invasive surgery, but full-scene semantic segmentation based on spectral imaging data and obtained during open surgery has received almost no attention to date. To address this gap in the literature, we are investigating the following research questions based on hyperspectral imaging (HSI) data of pigs acquired in an open surgery setting: (1) What is an adequate representation of HSI data for neural network-based fully automated organ segmentation, especially with respect to the spatial granularity of the data (pixels vs. superpixels vs. patches vs. full images)? (2) Is there a benefit of using HSI data compared to other modalities, namely RGB data and processed HSI data (e.g. tissue parameters like oxygenation), when performing semantic organ segmentation? According to a comprehensive validation study based on 506 HSI images from 20 pigs, annotated with a total of 19 classes, deep learning-based segmentation performance increases, consistently across modalities, with the spatial context of the input data. Unprocessed HSI data offers an advantage over RGB data or processed data from the camera provider, with the advantage increasing with decreasing size of the input to the neural network. Maximum performance (HSI applied to whole images) yielded a mean DSC of 0.90 ((standard deviation (SD)) 0.04), which is in the range of the inter-rater variability (DSC of 0.89 ((standard deviation (SD)) 0.07)). We conclude that HSI could become a powerful image modality for fully-automatic surgical scene understanding with many advantages over traditional imaging, including the ability to recover additional functional tissue information. Code and pre-trained models: https://github.com/IMSY-DKFZ/htc.

The first two authors (Silvia Seidlitz and Jan Sellner) contributed equally to this paper

References in corpus (7)

• The Medical Segmentation Decathlon
• A Spectral-Spatial-Dependent Global Learning Framework for Insufficient and Imbalanced Hyperspectral Image Classification
• Spatio-spectral classification of hyperspectral images for brain cancer detection during surgical operations
• In-Vivo Hyperspectral Human Brain Image Database for Brain Cancer Detection
• Quantifying the Carbon Emissions of Machine Learning
• Trends in deep learning for medical hyperspectral image analysis
• Common Limitations of Image Processing Metrics: A Picture Story

Cited by in corpus (13)

• Video-rate multispectral imaging in laparoscopic surgery: First-in-human application
• On-chip Hyperspectral Image Segmentation with Fully Convolutional Networks for Scene Understanding in Autonomous Driving
• Deep Learning for Pancreas Segmentation: a Systematic Review
• HSI-Drive v2.0: More Data for New Challenges in Scene Understanding for Autonomous Driving
• Hyperspectral Image Segmentation: A Preliminary Study on the Oral and Dental Spectral Image Database (ODSI-DB)
• Rapid Deployment of Domain-specific Hyperspectral Image Processors with Application to Autonomous Driving
• Deep intra-operative illumination calibration of hyperspectral cameras
• Automated Charge Transition Detection in Quantum Dot Charge Stability Diagrams
• OOD-SEG: Exploiting out-of-distribution detection techniques for learning image segmentation from sparse multi-class positive-only annotations
• SAMSA: Segment Anything Model Enhanced with Spectral Angles for Hyperspectral Interactive Medical Image Segmentation
• Tree-based Semantic Losses: Application to Sparsely-supervised Large Multi-class Hyperspectral Segmentation
• Label tree semantic losses for rich multi-class medical image segmentation
• First Investigation of Deep Learning for Intraoperative Gauze Segmentation in Minimally Invasive Abdominal Surgery