Segment Anything Model for Medical Images?
arXiv:2304.14660 · doi:10.1016/j.media.2023.103061
Abstract
The Segment Anything Model (SAM) is the first foundation model for general image segmentation. It has achieved impressive results on various natural image segmentation tasks. However, medical image segmentation (MIS) is more challenging because of the complex modalities, fine anatomical structures, uncertain and complex object boundaries, and wide-range object scales. To fully validate SAM's performance on medical data, we collected and sorted 53 open-source datasets and built a large medical segmentation dataset with 18 modalities, 84 objects, 125 object-modality paired targets, 1050K 2D images, and 6033K masks. We comprehensively analyzed different models and strategies on the so-called COSMOS 1050K dataset. Our findings mainly include the following: 1) SAM showed remarkable performance in some specific objects but was unstable, imperfect, or even totally failed in other situations. 2) SAM with the large ViT-H showed better overall performance than that with the small ViT-B. 3) SAM performed better with manual hints, especially box, than the Everything mode. 4) SAM could help human annotation with high labeling quality and less time. 5) SAM was sensitive to the randomness in the center point and tight box prompts, and may suffer from a serious performance drop. 6) SAM performed better than interactive methods with one or a few points, but will be outpaced as the number of points increases. 7) SAM's performance correlated to different factors, including boundary complexity, intensity differences, etc. 8) Finetuning the SAM on specific medical tasks could improve its average DICE performance by 4.39% and 6.68% for ViT-B and ViT-H, respectively. We hope that this comprehensive report can help researchers explore the potential of SAM applications in MIS, and guide how to appropriately use and develop SAM.
arXiv commentAccepted by Medical Image Analysis. 23 pages, 18 figures, 8 tables
References in corpus (28)
- Automated Design of Deep Learning Methods for Biomedical Image Segmentation
- The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions
- Segment Anything in Medical Images
- TotalSegmentator: robust segmentation of 104 anatomical structures in CT images
- Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challenge
- The Liver Tumor Segmentation Benchmark (LiTS)
- Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
- Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)
- CHAOS Challenge -- Combined (CT-MR) Healthy Abdominal Organ Segmentation
- Deep Learning for Segmentation using an Open Large-Scale Dataset in 2D Echocardiography
- Segment Anything Model for Medical Image Analysis: an Experimental Study
- A large annotated medical image dataset for the development and evaluation of segmentation algorithms
- MONAI: An open-source framework for deep learning in healthcare
- VerSe: A Vertebrae Labelling and Segmentation Benchmark for Multi-detector CT Images
- MILD-Net: Minimal Information Loss Dilated Network for Gland Instance Segmentation in Colon Histology Images
- The KiTS19 Challenge Data: 300 Kidney Tumor Cases with Clinical Context, CT Semantic Segmentations, and Surgical Outcomes
- Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation
- Segment Anything Is Not Always Perfect: An Investigation of SAM on Different Real-world Applications
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- WORD: A large scale dataset, benchmark and clinical applicable study for abdominal organ segmentation from CT image
- An automatic multi-tissue human fetal brain segmentation benchmark using the Fetal Tissue Annotation Dataset
- Multi-Site Infant Brain Segmentation Algorithms: The iSeg-2019 Challenge
- Segment Anything Model (SAM) for Digital Pathology: Assess Zero-shot Segmentation on Whole Slide Imaging
- SAM Struggles in Concealed Scenes -- Empirical Study on Segment Anything
- Semantic-SAM: Segment and Recognize Anything at Any Granularity
- Can SAM Segment Anything? When SAM Meets Camouflaged Object Detection
- SAMM (Segment Any Medical Model): A 3D Slicer Integration to SAM
- MEDIAR: Harmony of Data-Centric and Model-Centric for Multi-Modality Microscopy
Cited by in corpus (19)
- Segment Anything in Medical Images
- Segment Anything Model for Medical Image Analysis: an Experimental Study
- 3DSAM-adapter: Holistic adaptation of SAM from 2D to 3D for promptable tumor segmentation
- Artificial General Intelligence for Medical Imaging Analysis
- MIRAGE: Multimodal foundation model and benchmark for comprehensive retinal OCT image analysis
- One-Step Abductive Multi-Target Learning with Diverse Noisy Samples and Its Application to Tumour Segmentation for Breast Cancer
- Is SAM 2 Better than SAM in Medical Image Segmentation?
- From Classical Machine Learning to Emerging Foundation Models: Review on Multimodal Data Integration for Cancer Research
- Zero-Shot Segmentation of Eye Features Using the Segment Anything Model (SAM)
- NuSegDG: Integration of Heterogeneous Space and Gaussian Kernel for Domain-Generalized Nuclei Segmentation
- Tissue Concepts: supervised foundation models in computational pathology
- Extreme Cardiac MRI Analysis under Respiratory Motion: Results of the CMRxMotion Challenge
- Adapting SAM for Volumetric X-Ray Data-sets of Arbitrary Sizes
- Generalist Models in Medical Image Segmentation: A Survey and Performance Comparison with Task-Specific Approaches
- SUPER-Net: Trustworthy Image Segmentation via Uncertainty Propagation in Encoder-Decoder Networks
- Bayesian SegNet for Semantic Segmentation with Improved Interpretation of Microstructural Evolution During Irradiation of Materials
- A Comparison of Deep Learning Methods for Cell Detection in Digital Cytology
- Can Foundation Models Really Segment Tumors? A Benchmarking Odyssey in Lung CT Imaging
- OBSeg: Accurate and Fast Instance Segmentation Framework Using Segmentation Foundation Models with Oriented Bounding Box Prompts