BiomedParse: a biomedical foundation model for image parsing of everything everywhere all at once

Abstract

Biomedical image analysis is fundamental for biomedical discovery in cell biology, pathology, radiology, and many other biomedical domains. Holistic image analysis comprises interdependent subtasks such as segmentation, detection, and recognition of relevant objects. Here, we propose BiomedParse, a biomedical foundation model for imaging parsing that can jointly conduct segmentation, detection, and recognition for 82 object types across 9 imaging modalities. Through joint learning, we can improve accuracy for individual tasks and enable novel applications such as segmenting all relevant objects in an image through a text prompt, rather than requiring users to laboriously specify the bounding box for each object. We leveraged readily available natural-language labels or descriptions accompanying those datasets and use GPT-4 to harmonize the noisy, unstructured text information with established biomedical object ontologies. We created a large dataset comprising over six million triples of image, segmentation mask, and textual description. On image segmentation, we showed that BiomedParse is broadly applicable, outperforming state-of-the-art methods on 102,855 test image-mask-label triples across 9 imaging modalities (everything). On object detection, which aims to locate a specific object of interest, BiomedParse again attained state-of-the-art performance, especially on objects with irregular shapes (everywhere). On object recognition, which aims to identify all objects in a given image along with their semantic types, we showed that BiomedParse can simultaneously segment and label all biomedical objects in an image (all at once). In summary, BiomedParse is an all-in-one tool for biomedical image analysis by jointly solving segmentation, detection, and recognition for all major biomedical image modalities, paving the path for efficient and accurate image-based biomedical discovery.

Project page: https://aka.ms/biomedparse-project . Nat Methods (2024)

References in corpus (24)

• A Survey on Deep Learning in Medical Image Analysis
• Automated Design of Deep Learning Methods for Biomedical Image Segmentation
• TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
• Segment Anything in Medical Images
• Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
• Cell Detection with Star-convex Polygons
• Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation
• Medical Image Segmentation Using Deep Learning: A Survey
• Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection
• LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
• UNETR: Transformers for 3D Medical Image Segmentation
• AMOS: A Large-Scale Abdominal Multi-Organ Benchmark for Versatile Medical Image Segmentation
• PanNuke Dataset Extension, Insights and Baselines
• TransBTS: Multimodal Brain Tumor Segmentation Using Transformer
• Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks
• The Brain Tumor Segmentation (BraTS) Challenge 2023: Focus on Pediatrics (CBTN-CONNECT-DIPGR-ASNR-MICCAI BraTS-PEDs)
• CellViT: Vision Transformers for Precise Cell Segmentation and Classification
• Large-Vocabulary Segmentation for Medical Images with Text Prompts
• Distilling Large Language Models for Biomedical Knowledge Extraction: A Case Study on Adverse Drug Events
• Foundation Models for Biomedical Image Segmentation: A Survey
• BiomedJourney: Counterfactual Biomedical Image Generation by Instruction-Learning from Multimodal Patient Journeys
• OCTCube-M: A 3D multimodal optical coherence tomography foundation model for retinal and systemic diseases with cross-cohort and cross-device validation
• CircleSnake: Instance Segmentation with Circle Representation
• Scaling Up 3D Kernels with Bayesian Frequency Re-parameterization for Medical Image Segmentation

Cited by in corpus (3)

• Grounding DINO-US-SAM: Text-Prompted Multi-Organ Segmentation in Ultrasound with LoRA-Tuned Vision-Language Models
• Generalist Models in Medical Image Segmentation: A Survey and Performance Comparison with Task-Specific Approaches
• Clinical utility of foundation models in musculoskeletal MRI for biomarker fidelity and predictive outcomes