Segment Anything Model for Medical Images?
arXiv:2304.14660 · doi:10.1016/j.media.2023.103061
Abstract
The Segment Anything Model (SAM) is the first foundation model for general image segmentation. It has achieved impressive results on various natural image segmentation tasks. However, medical image segmentation (MIS) is more challenging because of the complex modalities, fine anatomical structures, uncertain and complex object boundaries, and wide-range object scales. To fully validate SAM's performance on medical data, we collected and sorted 53 open-source datasets and built a large medical segmentation dataset with 18 modalities, 84 objects, 125 object-modality paired targets, 1050K 2D images, and 6033K masks. We comprehensively analyzed different models and strategies on the so-called COSMOS 1050K dataset. Our findings mainly include the following: 1) SAM showed remarkable performance in some specific objects but was unstable, imperfect, or even totally failed in other situations. 2) SAM with the large ViT-H showed better overall performance than that with the small ViT-B. 3) SAM performed better with manual hints, especially box, than the Everything mode. 4) SAM could help human annotation with high labeling quality and less time. 5) SAM was sensitive to the randomness in the center point and tight box prompts, and may suffer from a serious performance drop. 6) SAM performed better than interactive methods with one or a few points, but will be outpaced as the number of points increases. 7) SAM's performance correlated to different factors, including boundary complexity, intensity differences, etc. 8) Finetuning the SAM on specific medical tasks could improve its average DICE performance by 4.39% and 6.68% for ViT-B and ViT-H, respectively. We hope that this comprehensive report can help researchers explore the potential of SAM applications in MIS, and guide how to appropriately use and develop SAM.
Accepted by Medical Image Analysis. 23 pages, 18 figures, 8 tables
References in corpus (14)
- Segment Anything in Medical Images
- Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
- Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)
- Segment Anything Model for Medical Image Analysis: an Experimental Study
- A large annotated medical image dataset for the development and evaluation of segmentation algorithms
- MONAI: An open-source framework for deep learning in healthcare
- Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation
- Segment Anything Is Not Always Perfect: An Investigation of SAM on Different Real-world Applications
- SAM Struggles in Concealed Scenes -- Empirical Study on Segment Anything
- Segment Anything Model (SAM) for Digital Pathology: Assess Zero-shot Segmentation on Whole Slide Imaging
- Semantic-SAM: Segment and Recognize Anything at Any Granularity
- Can SAM Segment Anything? When SAM Meets Camouflaged Object Detection
- SAMM (Segment Any Medical Model): A 3D Slicer Integration to SAM
- MEDIAR: Harmony of Data-Centric and Model-Centric for Multi-Modality Microscopy
Cited by in corpus (17)
- Segment Anything in Medical Images
- Segment Anything Model for Medical Image Analysis: an Experimental Study
- 3DSAM-adapter: Holistic adaptation of SAM from 2D to 3D for promptable tumor segmentation
- Artificial General Intelligence for Medical Imaging Analysis
- MIRAGE: Multimodal foundation model and benchmark for comprehensive retinal OCT image analysis
- Is SAM 2 Better than SAM in Medical Image Segmentation?
- From Classical Machine Learning to Emerging Foundation Models: Review on Multimodal Data Integration for Cancer Research
- Zero-Shot Segmentation of Eye Features Using the Segment Anything Model (SAM)
- NuSegDG: Integration of Heterogeneous Space and Gaussian Kernel for Domain-Generalized Nuclei Segmentation
- Tissue Concepts: supervised foundation models in computational pathology
- Generalist Models in Medical Image Segmentation: A Survey and Performance Comparison with Task-Specific Approaches
- Adapting SAM for Volumetric X-Ray Data-sets of Arbitrary Sizes
- Extreme Cardiac MRI Analysis under Respiratory Motion: Results of the CMRxMotion Challenge
- Bayesian SegNet for Semantic Segmentation with Improved Interpretation of Microstructural Evolution During Irradiation of Materials
- Can Foundation Models Really Segment Tumors? A Benchmarking Odyssey in Lung CT Imaging
- A Comparison of Deep Learning Methods for Cell Detection in Digital Cytology
- OBSeg: Accurate and Fast Instance Segmentation Framework Using Segmentation Foundation Models with Oriented Bounding Box Prompts