Frontiers in Intelligent Colonoscopy

Abstract

Colonoscopy is currently one of the most sensitive screening methods for colorectal cancer. This study investigates the frontiers of intelligent colonoscopy techniques and their prospective implications for multimodal medical applications. With this goal, we begin by assessing the current data-centric and model-centric landscapes through four tasks for colonoscopic scene perception, including classification, detection, segmentation, and vision-language understanding. This assessment enables us to identify domain-specific challenges and reveals that multimodal research in colonoscopy remains open for further exploration. To embrace the coming multimodal era, we establish three foundational initiatives: a large-scale multimodal instruction tuning dataset ColonINST, a colonoscopy-designed multimodal language model ColonGPT, and a multimodal benchmark. To facilitate ongoing monitoring of this rapidly evolving field, we provide a public website for the latest updates: https://github.com/ai4colonoscopy/IntelliScope.

[Work in progress] A comprehensive survey of intelligent colonoscopy in the multimodal era. [Updated Version V2] New training strategy for colonoscopy-specific multimodal language model

References in corpus (23)

• SSD: Single Shot MultiBox Detector
• Res2Net: A New Multi-scale Backbone Architecture
• Road Extraction by Deep Residual U-Net
• Segment Anything in Medical Images
• PVT v2: Improved Baselines with Pyramid Vision Transformer
• UACANet: Uncertainty Augmented Context Attention for Polyp Segmentation
• Deep Gradient Learning for Efficient Camouflaged Object Detection
• Uncertainty and Interpretability in Convolutional Neural Networks for Semantic Segmentation of Colorectal Polyps
• Video Polyp Segmentation: A Deep Learning Perspective
• Deep learning for detection and segmentation of artefact and disease instances in gastrointestinal endoscopy
• A multi-centre polyp detection and segmentation dataset for generalisability assessment
• A Survey on Long-Tailed Visual Recognition
• Progressively Normalized Self-Attention Network for Video Polyp Segmentation
• Advances in Deep Concealed Scene Understanding
• SinGAN-Seg: Synthetic training data generation for medical image segmentation
• Polyp Detection and Segmentation from Video Capsule Endoscopy: A Review
• A Survey on Deep Learning for Polyp Segmentation: Techniques, Challenges and Future Trends
• SliceMamba with Neural Architecture Search for Medical Image Segmentation
• Kvasir-VQA: A Text-Image Pair GI Tract Dataset
• Rethinking Polyp Segmentation from an Out-of-Distribution Perspective
• Self-Supervised Polyp Re-Identification in Colonoscopy
• Acquiring Weak Annotations for Tumor Localization in Temporal and Volumetric Data
• Deep Learning for Improved Polyp Detection from Synthetic Narrow-Band Imaging