VSE++: Improving Visual-Semantic Embeddings with Hard Negatives

Abstract

We present a new technique for learning visual-semantic embeddings for cross-modal retrieval. Inspired by hard negative mining, the use of hard negatives in structured prediction, and ranking loss functions, we introduce a simple change to common loss functions used for multi-modal embeddings. That, combined with fine-tuning and use of augmented data, yields significant gains in retrieval performance. We showcase our approach, VSE++, on MS-COCO and Flickr30K datasets, using ablation studies and comparisons with existing methods. On MS-COCO our approach outperforms state-of-the-art methods by 8.8% in caption retrieval and 11.3% in image retrieval (at R@1).

Accepted as spotlight presentation at British Machine Vision Conference (BMVC) 2018. Code: https://github.com/fartashf/vsepp

References in corpus (6)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Generative Adversarial Text to Image Synthesis
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Direct Optimization of Ranking Measures
• Order-Embeddings of Images and Language

Cited by in corpus (77)

• Learning Transferable Visual Models From Natural Language Supervision
• ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
• Data Augmentation using Random Image Cropping and Patching for Deep CNNs
• Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval
• Exploring a Fine-Grained Multiscale Method for Cross-Modal Remote Sensing Image Retrieval
• Adversarial Text-to-Image Synthesis: A Review
• Remote Sensing Cross-Modal Text-Image Retrieval Based on Global and Local Information
• Supervised Multimodal Bitransformers for Classifying Images and Text
• Learning Dual Semantic Relations with Graph Attention for Image-Text Matching
• Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
• Semantically Self-Aligned Network for Text-to-Image Part-aware Person Re-identification
• Reading-strategy Inspired Visual Representation Learning for Text-to-Video Retrieval
• Visually Grounded Neural Syntax Acquisition
• Discriminability objective for training descriptive captions
• Show, Recall, and Tell: Image Captioning with Recall Mechanism
• Efficient Token-Guided Image-Text Retrieval with Consistent Multimodal Contrastive Training
• Video Storytelling: Textual Summaries for Events
• Stacked Cross Attention for Image-Text Matching
• Plug-and-Play Regulators for Image-Text Matching
• Direction-Oriented Visual-semantic Embedding Model for Remote Sensing Image-text Retrieval
• Your Negative May not Be True Negative: Boosting Image-Text Matching with False Negative Elimination
• Beyond Visual Semantics: Exploring the Role of Scene Text in Image Understanding
• Look, Imagine and Match: Improving Textual-Visual Cross-Modal Retrieval with Generative Models
• Unifying Two-Stream Encoders with Transformers for Cross-Modal Retrieval
• On Distinctive Image Captioning via Comparing and Reweighting
• CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval
• Learning Visual Relation Priors for Image-Text Matching and Image Captioning with Neural Scene Graph Generators
• IMRAM: Iterative Matching with Recurrent Attention Memory for Cross-Modal Image-Text Retrieval
• Text-based Localization of Moments in a Video Corpus
• Learning to Evaluate Performance of Multi-modal Semantic Localization
• Cross-Lingual Cross-Modal Retrieval with Noise-Robust Learning
• Improving Referring Expression Grounding with Cross-modal Attention-guided Erasing
• Learnable Pillar-based Re-ranking for Image-Text Retrieval
• 3SHNet: Boosting Image-Sentence Retrieval via Visual Semantic-Spatial Self-Highlighting
• Masked Vision-Language Transformer in Fashion
• A Hierarchical Multi-Modal Encoder for Moment Localization in Video Corpus
• ICECAP: Information Concentrated Entity-aware Image Captioning
• Feature Re-Learning with Data Augmentation for Video Relevance Prediction
• Hierarchical Matching and Reasoning for Multi-Query Image Retrieval
• Kaleido-BERT: Vision-Language Pre-training on Fashion Domain
• DSSL: Deep Surroundings-person Separation Learning for Text-based Person Retrieval
• Learn to Understand Negation in Video Retrieval
• T2VLAD: Global-Local Sequence Alignment for Text-Video Retrieval
• Visual-Textual Association with Hardest and Semi-Hard Negative Pairs Mining for Person Search
• Step-Wise Hierarchical Alignment Network for Image-Text Matching
• Deep Multimodal Neural Architecture Search
• Probing Inter-modality: Visual Parsing with Self-Attention for Vision-Language Pre-training
• HANet: Hierarchical Alignment Networks for Video-Text Retrieval
• Fine-Grained Image Captioning with Global-Local Discriminative Objective
• Open-Edit: Open-Domain Image Manipulation with Open-Vocabulary Instructions
• Contextual Lensing of Universal Sentence Representations
• Graceful Degradation and Related Fields
• TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignment
• Team RUC_AIM3 Technical Report at Activitynet 2020 Task 2: Exploring Sequential Events Detection for Dense Video Captioning
• HUSE: Hierarchical Universal Semantic Embeddings
• A Graph-Based Framework to Bridge Movies and Synopses
• Adaptive Offline Quintuplet Loss for Image-Text Matching
• A Multi-level Alignment Training Scheme for Video-and-Language Grounding
• FiCo-ITR: bridging fine-grained and coarse-grained image-text retrieval for comparative performance analysis
• Reply with Sticker: New Dataset and Model for Sticker Retrieval
• ASBERT: Siamese and Triplet network embedding for open question answering
• MCEN: Bridging Cross-Modal Gap between Cooking Recipes and Dish Images with Latent Variable Model
• Lessons learned in multilingual grounded language learning
• Beyond the Deep Metric Learning: Enhance the Cross-Modal Matching with Adversarial Discriminative Domain Regularization
• HNC: Leveraging Hard Negative Captions towards Models with Fine-Grained Visual-Linguistic Comprehension Capabilities
• Migration-Related Semantic Concepts for the Retrieval of Relevant Video Content
• An Unsupervised Sampling Approach for Image-Sentence Matching Using Document-Level Structural Information
• Multiple Visual-Semantic Embedding for Video Retrieval from Query Sentence
• Multimodal Information Retrieval for Open World with Edit Distance Weak Supervision
• YouMakeup VQA Challenge: Towards Fine-grained Action Understanding in Domain-Specific Videos
• Deep Unified Multimodal Embeddings for Understanding both Content and Users in Social Media Networks
• Show, Translate and Tell
• Bootstrapping Disjoint Datasets for Multilingual Multimodal Representation Learning
• Semi-supervised Visual Feature Integration for Pre-trained Language Models
• T-EMDE: Sketching-based global similarity for cross-modal retrieval
• ADNet: Temporal Anomaly Detection in Surveillance Videos
• Integrating Temporal and Spatial Attentions for VATEX Video Captioning Challenge 2019