Collaborating Foundation Models for Domain Generalized Semantic Segmentation

Abstract

Domain Generalized Semantic Segmentation (DGSS) deals with training a model on a labeled source domain with the aim of generalizing to unseen domains during inference. Existing DGSS methods typically effectuate robust features by means of Domain Randomization (DR). Such an approach is often limited as it can only account for style diversification and not content. In this work, we take an orthogonal approach to DGSS and propose to use an assembly of CoLlaborative FOUndation models for Domain Generalized Semantic Segmentation (CLOUDS). In detail, CLOUDS is a framework that integrates FMs of various kinds: (i) CLIP backbone for its robust feature representation, (ii) generative models to diversify the content, thereby covering various modes of the possible target distribution, and (iii) Segment Anything Model (SAM) for iteratively refining the predictions of the segmentation model. Extensive experiments show that our CLOUDS excels in adapting from synthetic to real DGSS benchmarks and under varying weather conditions, notably outperforming prior methods by 5.6% and 6.7% on averaged miou, respectively. The code is available at : https://github.com/yasserben/CLOUDS

https://github.com/yasserben/CLOUDS ; Accepted to CVPR 2024

References in corpus (26)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Generative Adversarial Networks: An Overview
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Hierarchical Text-Conditional Image Generation with CLIP Latents
• On the Opportunities and Risks of Foundation Models
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• LAION-5B: An open large-scale dataset for training next generation image-text models
• Domain Generalization via Invariant Feature Representation
• Segment Anything
• Generalizing to Unseen Domains via Adversarial Data Augmentation
• Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection
• Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation
• Few-Shot Adversarial Domain Adaptation
• Computer-Vision Benchmark Segment-Anything Model (SAM) in Medical Images: Accuracy in 12 Datasets
• Semantic-SAM: Segment and Recognize Anything at Any Granularity
• Adversarial Style Augmentation for Domain Generalized Urban-Scene Segmentation
• Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP
• Diffusion Models for Open-Vocabulary Segmentation
• Deep learning universal crater detection using Segment Anything Model (SAM)
• Prompting Diffusion Representations for Cross-Domain Semantic Segmentation
• CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation
• PromptStyler: Prompt-driven Style Generation for Source-free Domain Generalization
• Source-Free Open Compound Domain Adaptation in Semantic Segmentation
• Texture Learning Domain Randomization for Domain Generalized Segmentation
• Generalization by Adaptation: Diffusion-Based Domain Extension for Domain-Generalized Semantic Segmentation
• Domain Generalization via Balancing Training Difficulty and Model Capability