Tracking and Mapping in Medical Computer Vision: A Review
arXiv:2310.11475 · doi:10.1016/j.media.2024.103131
Abstract
As computer vision algorithms increase in capability, their applications in clinical systems will become more pervasive. These applications include: diagnostics, such as colonoscopy and bronchoscopy; guiding biopsies, minimally invasive interventions, and surgery; automating instrument motion; and providing image guidance using pre-operative scans. Many of these applications depend on the specific visual nature of medical scenes and require designing algorithms to perform in this environment. In this review, we provide an update to the field of camera-based tracking and scene mapping in surgery and diagnostics in medical computer vision. We begin with describing our review process, which results in a final list of 515 papers that we cover. We then give a high-level summary of the state of the art and provide relevant background for those who need tracking and mapping for their clinical applications. After which, we review datasets provided in the field and the clinical needs that motivate their design. Then, we delve into the algorithmic side, and summarize recent developments. This summary should be especially useful for algorithm designers and to those looking to understand the capability of off-the-shelf methods. We maintain focus on algorithms for deformable environments while also reviewing the essential building blocks in rigid tracking and mapping since there is a large amount of crossover in methods. With the field summarized, we discuss the current state of the tracking and mapping methods along with needs for future algorithms, needs for quantification, and the viability of clinical applications. We then provide some research directions and questions. We conclude that new methods need to be designed or combined to support clinical applications in deformable environments, and more focus needs to be put into collecting datasets for training and evaluation.
36 pages, 17 figures. To appear in Medical Image Analysis
References in corpus (31)
- ORB-SLAM: a Versatile and Accurate Monocular SLAM System
- Instant Neural Graphics Primitives with a Multiresolution Hash Encoding
- LoRA: Low-Rank Adaptation of Large Language Models
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- DINOv2: Learning Robust Visual Features without Supervision
- Metrics reloaded: Recommendations for image analysis validation
- Image Matching across Wide Baselines: From Paper to Practice
- BA-Net: Dense Bundle Adjustment Network
- Colonoscopy 3D Video Dataset with Paired Depth from 2D-3D Registration
- SuPer: A Surgical Perception Framework for Endoscopic Tissue Manipulation with Surgical Robotics
- PIN-SLAM: LiDAR SLAM Using a Point-Based Implicit Neural Representation for Achieving Global Map Consistency
- EndoMapper dataset of complete calibrated endoscopy procedures
- SERV-CT: A disparity dataset from CT for validation of endoscopic 3D reconstruction
- A Non-Rigid Map Fusion-Based RGB-Depth SLAM Method for Endoscopic Capsule Robots
- Deep Declarative Networks: A New Hope
- Stereo Dense Scene Reconstruction and Accurate Localization for Learning-Based Navigation of Laparoscope in Minimally Invasive Surgery
- A Tale of Two Features: Stable Diffusion Complements DINO for Zero-Shot Semantic Correspondence
- Bimodal Camera Pose Prediction for Endoscopy
- Segment Anything Meets Point Tracking
- EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision
- EndoGaussian: Real-time Gaussian Splatting for Dynamic Endoscopic Scene Reconstruction
- MITI: SLAM Benchmark for Laparoscopic Surgery
- Intrinsic Dynamic Shape Prior for Fast, Sequential and Dense Non-Rigid Structure from Motion with Detection of Temporally-Disjoint Rigidity
- DynPoint: Dynamic Neural Point For View Synthesis
- 3D endoscopic depth estimation using 3D surface-aware constraints
- EndoGS: Deformable Endoscopic Tissues Reconstruction with Gaussian Splatting
- Depth Anything in Medical Images: A Comparative Study
- Endo-4DGS: Endoscopic Monocular Scene Reconstruction with 4D Gaussian Splatting
- EndoGaussians: Single View Dynamic Gaussian Splatting for Deformable Endoscopic Tissues Reconstruction
- BASED: Bundle-Adjusting Surgical Endoscopic Dynamic Video Reconstruction using Neural Radiance Fields
- Dense 3D Reconstruction Through Lidar: A Comparative Study on Ex-vivo Porcine Tissue