Query2Label: A Simple Transformer Way to Multi-Label Classification

Abstract

This paper presents a simple and effective approach to solving the multi-label classification problem. The proposed approach leverages Transformer decoders to query the existence of a class label. The use of Transformer is rooted in the need of extracting local discriminative features adaptively for different labels, which is a strongly desired property due to the existence of multiple objects in one image. The built-in cross-attention module in the Transformer decoder offers an effective way to use label embeddings as queries to probe and pool class-related features from a feature map computed by a vision backbone for subsequent binary classifications. Compared with prior works, the new framework is simple, using standard Transformers and vision backbones, and effective, consistently outperforming all previous works on five multi-label classification data sets, including MS-COCO, PASCAL VOC, NUS-WIDE, and Visual Genome. Particularly, we establish $91.3\%$ mAP on MS-COCO. We hope its compact structure, simple implementation, and superior performance serve as a strong baseline for multi-label classification tasks and future studies. The code will be available soon at https://github.com/SlongLiu/query2labels.

References in corpus (14)

• Adam: A Method for Stochastic Optimization
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• A Survey on Visual Transformer
• Transformers in Vision: A Survey
• Improved Regularization of Convolutional Neural Networks with Cutout
• Deformable DETR: Deformable Transformers for End-to-End Object Detection
• CNN: Single-label to Multi-label
• A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay
• CvT: Introducing Convolutions to Vision Transformers
• Pre-Trained Image Processing Transformer
• Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks
• Asymmetric Loss For Multi-Label Classification
• Spatial Transformer Networks
• Chest X-rays Classification: A Multi-Label and Fine-Grained Problem

Cited by in corpus (9)

• Attention Mechanisms in Computer Vision: A Survey
• Multi-Label Knowledge Distillation
• LT-ViT: A Vision Transformer for multi-label Chest X-ray classification
• Rethinking Crowdsourcing Annotation: Partial Annotation with Salient Labels for Multi-Label Image Classification
• A vector quantized masked autoencoder for audiovisual speech emotion recognition
• IDEA: Increasing Text Diversity via Online Multi-Label Recognition for Vision-Language Pre-training
• Enhancing Classification with Hierarchical Scalable Query on Fusion Transformer
• FiCo-ITR: bridging fine-grained and coarse-grained image-text retrieval for comparative performance analysis
• MC-SSL0.0: Towards Multi-Concept Self-Supervised Learning