SIGN: Spatial-information Incorporated Generative Network for Generalized Zero-shot Semantic Segmentation

Abstract

Unlike conventional zero-shot classification, zero-shot semantic segmentation predicts a class label at the pixel level instead of the image level. When solving zero-shot semantic segmentation problems, the need for pixel-level prediction with surrounding context motivates us to incorporate spatial information using positional encoding. We improve standard positional encoding by introducing the concept of Relative Positional Encoding, which integrates spatial information at the feature level and can handle arbitrary image sizes. Furthermore, while self-training is widely used in zero-shot semantic segmentation to generate pseudo-labels, we propose a new knowledge-distillation-inspired self-training strategy, namely Annealed Self-Training, which can automatically assign different importance to pseudo-labels to improve performance. We systematically study the proposed Relative Positional Encoding and Annealed Self-Training in a comprehensive experimental evaluation, and our empirical results confirm the effectiveness of our method on three benchmark datasets.

arXiv commentAccepted in ICCV 2021

References in corpus (13)

• Distributed Representations of Words and Phrases and their Compositionality
• Auto-Encoding Variational Bayes
• Distilling the Knowledge in a Neural Network
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Zero-Shot Text-to-Image Generation
• Zero-Shot Learning Through Cross-Modal Transfer
• Label-Embedding for Image Classification
• Zero-Shot Learning by Convex Combination of Semantic Embeddings
• Generative Moment Matching Networks
• Polarity Loss for Zero-shot Object Detection
• Zero-Shot Object Detection by Hybrid Region Embedding
• Improve Transformer Models with Better Relative Position Embeddings