Sam2Rad: A Segmentation Model for Medical Images with Learnable Prompts
arXiv:2409.06821 · doi:10.1016/j.compbiomed.2025.109725
Abstract
Foundation models like the segment anything model require high-quality manual prompts for medical image segmentation, which is time-consuming and requires expertise. SAM and its variants often fail to segment structures in ultrasound (US) images due to domain shift. We propose Sam2Rad, a prompt learning approach to adapt SAM and its variants for US bone segmentation without human prompts. It introduces a prompt predictor network (PPN) with a cross-attention module to predict prompt embeddings from image encoder features. PPN outputs bounding box and mask prompts, and 256-dimensional embeddings for regions of interest. The framework allows optional manual prompting and can be trained end-to-end using parameter-efficient fine-tuning (PEFT). Sam2Rad was tested on 3 musculoskeletal US datasets: wrist (3822 images), rotator cuff (1605 images), and hip (4849 images). It improved performance across all datasets without manual prompts, increasing Dice scores by 2-7% for hip/wrist and up to 33% for shoulder data. Sam2Rad can be trained with as few as 10 labeled images and is compatible with any SAM architecture for automatic segmentation.
References in corpus (22)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Decoupled Weight Decay Regularization
- Attention U-Net: Learning Where to Look for the Pancreas
- TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
- Segment Anything in Medical Images
- LoRA: Low-Rank Adaptation of Large Language Models
- Scaling Laws for Neural Language Models
- Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation
- Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation
- Faster Segment Anything: Towards Lightweight SAM for Mobile Applications
- Customized Segment Anything Model for Medical Image Segmentation
- Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks
- SAM.MD: Zero-shot medical image segmentation capabilities of the Segment Anything Model
- Personalize Segment Anything Model with One Shot
- A Constructive Prediction of the Generalization Error Across Scales
- SAM Fails to Segment Anything? -- SAM-Adapter: Adapting SAM in Underperformed Scenes: Camouflage, Shadow, Medical Image Segmentation, and More
- AutoSAM: Adapting SAM to Medical Images by Overloading the Prompt Encoder
- Segment Anything Model (SAM) Enhanced Pseudo Labels for Weakly Supervised Semantic Segmentation
- Few-Shot Panoptic Segmentation With Foundation Models
- Matcher: Segment Anything with One Shot Using All-Purpose Feature Matching
- Benchmarking Zero-Shot Robustness of Multimodal Foundation Models: A Pilot Study
- MaskSAM: Towards Auto-prompt SAM with Mask Classification for Volumetric Medical Image Segmentation