Language-driven Semantic Segmentation
arXiv:2201.03546
Abstract
We present LSeg, a novel model for language-driven semantic image segmentation. LSeg uses a text encoder to compute embeddings of descriptive input labels (e.g., "grass" or "building") together with a transformer-based image encoder that computes dense per-pixel embeddings of the input image. The image encoder is trained with a contrastive objective to align pixel embeddings to the text embedding of the corresponding semantic class. The text embeddings provide a flexible label representation in which semantically similar labels map to similar regions in the embedding space (e.g., "cat" and "furry"). This allows LSeg to generalize to previously unseen categories at test time, without retraining or even requiring a single additional training sample. We demonstrate that our approach achieves highly competitive zero-shot performance compared to existing zero- and few-shot semantic segmentation methods, and even matches the accuracy of traditional segmentation algorithms when a fixed label set is provided. Code and demo are available at https://github.com/isl-org/lang-seg.
arXiv commentICLR 2022
Cited by in corpus (6)
- L3MVN: Leveraging Large Language Models for Visual Target Navigation
- Iterative Few-shot Semantic Segmentation from Image Label Text
- Blended-NeRF: Zero-Shot Object Generation and Blending in Existing Neural Radiance Fields
- OV-MAP: Open-Vocabulary Zero-Shot 3D Instance Segmentation Map for Robots
- PØDA: Prompt-driven Zero-shot Domain Adaptation
- NeuroSMPC: A Neural Network guided Sampling Based MPC for On-Road Autonomous Driving