Similar Image Search for Histopathology: SMILY

Abstract

The increasing availability of large institutional and public histopathology image datasets is enabling the searching of these datasets for diagnosis, research, and education. Though these datasets typically have associated metadata such as diagnosis or clinical notes, even carefully curated datasets rarely contain annotations of the location of regions of interest on each image. Because pathology images are extremely large (up to 100,000 pixels in each dimension), further laborious visual search of each image may be needed to find the feature of interest. In this paper, we introduce a deep learning based reverse image search tool for histopathology images: Similar Medical Images Like Yours (SMILY). We assessed SMILY's ability to retrieve search results in two ways: using pathologist-provided annotations, and via prospective studies where pathologists evaluated the quality of SMILY search results. As a negative control in the second evaluation, pathologists were blinded to whether search results were retrieved by SMILY or randomly. In both types of assessments, SMILY was able to retrieve search results with similar histologic features, organ site, and prostate cancer Gleason grade compared with the original query. SMILY may be a useful general-purpose tool in the pathologist's arsenal, to improve the efficiency of searching large archives of histopathology images, without the need to develop and implement specific tools for each application.

arXiv comment23 Pages with 6 figures and 3 tables. The file also has 6 pages of supplemental material. Improved figure resolution, edited metadata

References in corpus (2)

• Assessment of algorithms for mitosis detection in breast cancer histopathology images
• Predicting breast tumor proliferation from whole-slide images: the TUPAC16 challenge

Cited by in corpus (14)

• Microscope 2.0: An Augmented Reality Microscope with Real-time Artificial Intelligence Integration
• Artificial Intelligence for Digital and Computational Pathology
• Interpretable Survival Prediction for Colorectal Cancer using Deep Learning
• Deep Model with Siamese Network for Viability and Necrosis Tumor Assessment in Osteosarcoma
• Lessons Learned from Designing an AI-Enabled Diagnosis Tool for Pathologists
• Evaluating Pre-trained Convolutional Neural Networks and Foundation Models as Feature Extractors for Content-based Medical Image Retrieval
• Evolutionary Computation in Action: Feature Selection for Deep Embedding Spaces of Gigapixel Pathology Images
• A vision transformer-based framework for knowledge transfer from multi-modal to mono-modal lymphoma subtyping models
• Analysis and Validation of Image Search Engines in Histopathology
• Towards More Transparent and Accurate Cancer Diagnosis with an Unsupervised CAE Approach
• Surgment: Segmentation-enabled Semantic Search and Creation of Visual Question and Feedback to Support Video-Based Surgery Learning
• Pan-cancer Histopathology WSI Pre-training with Position-aware Masked Autoencoder
• Immunohistochemistry Biomarkers-Guided Image Search for Histopathology
• From slides to AI-ready maps: Standardized multi-layer tissue maps as metadata for artificial intelligence in digital pathology