Learning Deep Representations of Fine-grained Visual Descriptions

Abstract

State-of-the-art methods for zero-shot visual recognition formulate learning as a joint embedding problem of images and side information. In these formulations the current best complement to visual features are attributes: manually encoded vectors describing shared characteristics among categories. Despite good performance, attributes have limitations: (1) finer-grained recognition requires commensurately more attributes, and (2) attributes do not provide a natural language interface. We propose to overcome these limitations by training neural language models from scratch; i.e. without pre-training and only consuming words and characters. Our proposed models train end-to-end to align with the fine-grained and category-specific content of images. Natural language provides a flexible and compact way of encoding only the salient visual aspects for distinguishing categories. By training on raw text, our model can do inference on raw text as well, providing humans a familiar mode both for annotation and retrieval. Our model achieves strong performance on zero-shot text-based image retrieval and significantly outperforms the attribute-based state-of-the-art for zero-shot classification on the Caltech UCSD Birds 200-2011 dataset.

CVPR 2016

References in corpus (9)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Distributed Representations of Words and Phrases and their Compositionality
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• Conditional Random Fields as Recurrent Neural Networks
• Zero-Shot Learning Through Cross-Modal Transfer
• Label-Embedding for Image Classification
• Zero-Shot Learning by Convex Combination of Semantic Embeddings
• Transductive Multi-view Zero-Shot Learning

Cited by in corpus (26)

• Adversarial Text-to-Image Synthesis: A Review
• An Introduction to Image Synthesis with Generative Adversarial Nets
• Fine-grained Visual-textual Representation Learning
• Person Search with Natural Language Description
• Transductive Zero-Shot Learning with Visual Structure Constraint
• CrossATNet - A Novel Cross-Attention Based Framework for Sketch-Based Image Retrieval
• Survey on Deep Multi-modal Data Analytics: Collaboration, Rivalry and Fusion
• Embedding Deep Networks into Visual Explanations
• Intrinsic Relationship Reasoning for Small Object Detection
• Learning the Redundancy-free Features for Generalized Zero-Shot Object Recognition
• DSSL: Deep Surroundings-person Separation Learning for Text-based Person Retrieval
• Improving Description-based Person Re-identification by Multi-granularity Image-text Alignments
• KG-GAN: Knowledge-Guided Generative Adversarial Networks
• Deep Cross-Modal Audio-Visual Generation
• Domain segmentation and adjustment for generalized zero-shot learning
• Classifier and Exemplar Synthesis for Zero-Shot Learning
• Alleviating Feature Confusion for Generative Zero-shot Learning
• What Can Knowledge Bring to Machine Learning? -- A Survey of Low-shot Learning for Structured Data
• TreeGAN: Incorporating Class Hierarchy into Image Generation
• CIZSL++: Creativity Inspired Generative Zero-Shot Learning
• Text-to-image Synthesis via Symmetrical Distillation Networks
• Bias-Awareness for Zero-Shot Learning the Seen and Unseen
• RGB-D Salient Object Detection Based on Discriminative Cross-modal Transfer Learning
• Mitigating Generation Shifts for Generalized Zero-Shot Learning
• DFS: A Diverse Feature Synthesis Model for Generalized Zero-Shot Learning
• Synthesizing Novel Pairs of Image and Text