SurgiTrack: Fine-Grained Multi-Class Multi-Tool Tracking in Surgical Videos
arXiv:2405.20333 · doi:10.1016/j.media.2024.103438
Abstract
Accurate tool tracking is essential for the success of computer-assisted intervention. Previous efforts often modeled tool trajectories rigidly, overlooking the dynamic nature of surgical procedures, especially tracking scenarios like out-of-body and out-of-camera views. Addressing this limitation, the new CholecTrack20 dataset provides detailed labels that account for multiple tool trajectories in three perspectives: (1) intraoperative, (2) intracorporeal, and (3) visibility, representing the different types of temporal duration of tool tracks. These fine-grained labels enhance tracking flexibility but also increase the task complexity. Re-identifying tools after occlusion or re-insertion into the body remains challenging due to high visual similarity, especially among tools of the same category. This work recognizes the critical role of the tool operators in distinguishing tool track instances, especially those belonging to the same tool category. The operators' information are however not explicitly captured in surgical videos. We therefore propose SurgiTrack, a novel deep learning method that leverages YOLOv7 for precise tool detection and employs an attention mechanism to model the originating direction of the tools, as a proxy to their operators, for tool re-identification. To handle diverse tool trajectory perspectives, SurgiTrack employs a harmonizing bipartite matching graph, minimizing conflicts and ensuring accurate tool identity association. Experimental results on CholecTrack20 demonstrate SurgiTrack's effectiveness, outperforming baselines and state-of-the-art methods with real-time inference capability. This work sets a new standard in surgical tool tracking, providing dynamic trajectories for more adaptable and precise assistance in minimally invasive surgeries.
arXiv comment15 pages, 7 figures, 7 tables, 1 video. Supplementary video available at: https://vimeo.com/951853260 . Article published in Medical Image Analysis Journal 2025
References in corpus (12)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Neural Machine Translation by Jointly Learning to Align and Translate
- FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking
- HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking
- Tracking without bells and whistles
- MOT20: A benchmark for multi object tracking in crowded scenes
- CrowdHuman: A Benchmark for Detecting Human in a Crowd
- TransTrack: Multiple Object Tracking with Transformer
- BoT-SORT: Robust Associations Multi-Pedestrian Tracking
- Weakly Supervised Convolutional LSTM Approach for Tool Tracking in Laparoscopic Videos
- Methods and datasets for segmentation of minimally invasive surgical instruments in endoscopic images and videos: A review of the state of the art
- CholecTrack20: A Multi-Perspective Tracking Dataset for Surgical Tools