Evaluating Pre-trained Convolutional Neural Networks and Foundation Models as Feature Extractors for Content-based Medical Image Retrieval

Abstract

Medical image retrieval refers to the task of finding similar images for given query images in a database, with applications such as diagnosis support. While traditional medical image retrieval relied on clinical metadata, content-based medical image retrieval (CBMIR) depends on image features, which can be extracted automatically or semi-automatically. Many approaches have been proposed for CBMIR, and among them, using pre-trained convolutional neural networks (CNNs) is a widely utilized approach. However, considering the recent advances in the development of foundation models for various computer vision tasks, their application for CBMIR can also be investigated. In this study, we used several pre-trained feature extractors from well-known pre-trained CNNs and pre-trained foundation models and investigated the CBMIR performance on eight types of two-dimensional (2D) and three-dimensional (3D) medical images. Furthermore, we investigated the effect of image size on the CBMIR performance. Our results show that, overall, for the 2D datasets, foundation models deliver superior performance by a large margin compared to CNNs, with the general-purpose self-supervised model for computational pathology (UNI) providing the best overall performance across all datasets and image sizes. For 3D datasets, CNNs and foundation models deliver more competitive performance, with contrastive learning from captions for histopathology model (CONCH) achieving the best overall performance. Moreover, our findings confirm that while using larger image sizes (especially for 2D datasets) yields slightly better performance, competitive CBMIR performance can still be achieved even with smaller image sizes. Our codes to reproduce the results are available at: https://github.com/masih4/MedImageRetrieval.

37 pages

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Automated Design of Deep Learning Methods for Biomedical Image Segmentation
• Neural Architecture Search with Reinforcement Learning
• The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions
• Segment Anything in Medical Images
• Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
• BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining
• DINOv2: Learning Robust Visual Features without Supervision
• MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification
• CoCa: Contrastive Captioners are Image-Text Foundation Models
• iBOT: Image BERT Pre-Training with Online Tokenizer
• Similar Image Search for Histopathology: SMILY
• Accelerator-aware Neural Network Design using AutoML
• Leveraging Foundation Models for Content-Based Image Retrieval in Radiology
• Leveraging Medical Foundation Model Features in Graph Neural Network-Based Retrieval of Breast Histopathology Images
• Content-Based Image Retrieval for Multi-Class Volumetric Radiology Images: A Benchmark Study

Cited by in corpus (1)

• Leveraging Medical Foundation Model Features in Graph Neural Network-Based Retrieval of Breast Histopathology Images