Fine-grained Visual-textual Representation Learning

Abstract

Fine-grained visual categorization is to recognize hundreds of subcategories belonging to the same basic-level category, which is a highly challenging task due to the quite subtle and local visual distinctions among similar subcategories. Most existing methods generally learn part detectors to discover discriminative regions for better categorization performance. However, not all parts are beneficial and indispensable for visual categorization, and the setting of part detector number heavily relies on prior knowledge as well as experimental validation. As is known to all, when we describe the object of an image via textual descriptions, we mainly focus on the pivotal characteristics, and rarely pay attention to common characteristics as well as the background areas. This is an involuntary transfer from human visual attention to textual attention, which leads to the fact that textual attention tells us how many and which parts are discriminative and significant to categorization. So textual attention could help us to discover visual attention in image. Inspired by this, we propose a fine-grained visual-textual representation learning (VTRL) approach, and its main contributions are: (1) Fine-grained visual-textual pattern mining devotes to discovering discriminative visual-textual pairwise information for boosting categorization performance through jointly modeling vision and text with generative adversarial networks (GANs), which automatically and adaptively discovers discriminative parts. (2) Visual-textual representation learning jointly combines visual and textual information, which preserves the intra-modality and inter-modality information to generate complementary fine-grained representation, as well as further improves categorization performance.

12 pages, accepted by IEEE Transactions on Circuits and Systems for Video Technology (TCSVT)

References in corpus (6)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Generative Adversarial Text to Image Synthesis
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• Bird Species Categorization Using Pose Normalized Deep Convolutional Nets
• Learning Deep Representations of Fine-grained Visual Descriptions
• Fine-graind Image Classification via Combining Vision and Language

Cited by in corpus (7)

• Learning Dual Semantic Relations with Graph Attention for Image-Text Matching
• CLIP-Art: Contrastive Pre-training for Fine-Grained Art Classification
• TOAN: Target-Oriented Alignment Network for Fine-Grained Image Categorization with Few Labeled Samples
• Fine-Grained Fashion Similarity Prediction by Attribute-Specific Embedding Learning
• 3D Face Anti-spoofing with Factorized Bilinear Coding
• Dependency Decomposition and a Reject Option for Explainable Models
• Unsupervised discovery of Interpretable Visual Concepts