3DSAM-adapter: Holistic adaptation of SAM from 2D to 3D for promptable tumor segmentation
arXiv:2306.13465 · doi:10.1016/j.media.2024.103324
Abstract
Despite that the segment anything model (SAM) achieved impressive results on general-purpose semantic segmentation with strong generalization ability on daily images, its demonstrated performance on medical image segmentation is less precise and not stable, especially when dealing with tumor segmentation tasks that involve objects of small sizes, irregular shapes, and low contrast. Notably, the original SAM architecture is designed for 2D natural images, therefore would not be able to extract the 3D spatial information from volumetric medical data effectively. In this paper, we propose a novel adaptation method for transferring SAM from 2D to 3D for promptable medical image segmentation. Through a holistically designed scheme for architecture modification, we transfer the SAM to support volumetric inputs while retaining the majority of its pre-trained parameters for reuse. The fine-tuning process is conducted in a parameter-efficient manner, wherein most of the pre-trained parameters remain frozen, and only a few lightweight spatial adapters are introduced and tuned. Regardless of the domain gap between natural and medical data and the disparity in the spatial arrangement between 2D and 3D, the transformer trained on natural images can effectively capture the spatial patterns present in volumetric medical images with only lightweight adaptations. We conduct experiments on four open-source tumor segmentation datasets, and with a single click prompt, our model can outperform domain state-of-the-art medical image segmentation models on 3 out of 4 tasks, specifically by 8.25%, 29.87%, and 10.11% for kidney tumor, pancreas tumor, colon cancer segmentation, and achieve similar performance for liver tumor segmentation. We also compare our adaptation method with existing popular adapters, and observed significant performance improvement on most datasets.
14 pages, 6 figures, 5 tables
References in corpus (22)
- Bootstrap your own latent: A new approach to self-supervised Learning
- Segment Anything in Medical Images
- LoRA: Low-Rank Adaptation of Large Language Models
- Hierarchical Text-Conditional Image Generation with CLIP Latents
- Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
- Segment Anything Model for Medical Image Analysis: an Experimental Study
- Capabilities of GPT-4 on Medical Challenge Problems
- Segment Anything Model for Medical Images?
- Early Convolutions Help Transformers See Better
- Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation
- Segment Anything Is Not Always Perfect: An Investigation of SAM on Different Real-world Applications
- AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
- Segment Everything Everywhere All at Once
- 3D UX-Net: A Large Kernel Volumetric ConvNet Modernizing Hierarchical Transformer for Medical Image Segmentation
- SegGPT: Segmenting Everything In Context
- AIM: Adapting Image Models for Efficient Video Action Recognition
- Interactive segmentation of medical images through fully convolutional neural networks
- UPL-SFDA: Uncertainty-aware Pseudo Label Guided Source-Free Domain Adaptation for Medical Image Segmentation
- SAM on Medical Images: A Comprehensive Study on Three Prompt Modes
- UNETR++: Delving into Efficient and Accurate 3D Medical Image Segmentation
- How to Efficiently Adapt Large Segmentation Model(SAM) to Medical Images
- nnSAM: Plug-and-play Segment Anything Model Improves nnUNet Performance