Finding beans in burgers: Deep semantic-visual embedding with localization

Abstract

Several works have proposed to learn a two-path neural network that maps images and texts, respectively, to a same shared Euclidean space where geometry captures useful semantic relationships. Such a multi-modal embedding can be trained and used for various tasks, notably image captioning. In the present work, we introduce a new architecture of this type, with a visual path that leverages recent space-aware pooling mechanisms. Combined with a textual path which is jointly trained from scratch, our semantic-visual embedding offers a versatile model. Once trained under the supervision of captioned images, it yields new state-of-the-art performance on cross-modal retrieval. It also allows the localization of new concepts from the embedding space into any input image, delivering state-of-the-art result on the visual grounding of phrases.

arXiv commentAccepted to CVPR2018

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Efficient Estimation of Word Representations in Vector Space
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• R-FCN: Object Detection via Region-based Fully Convolutional Networks
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Skip-Thought Vectors

Cited by in corpus (15)

• Semantics for Robotic Mapping, Perception and Interaction: A Survey
• Structured Multi-modal Feature Embedding and Alignment for Image-Sentence Retrieval
• Universal Multimodal Representation for Language Understanding
• Image search using multilingual texts: a cross-modal learning approach between image and text
• Sub-Instruction Aware Vision-and-Language Navigation
• Task-Adaptive Neural Network Search with Meta-Contrastive Learning
• UniVSE: Robust Visual Semantic Embeddings via Structured Semantic Representations
• MHSAN: Multi-Head Self-Attention Network for Visual Semantic Embedding
• MCEN: Bridging Cross-Modal Gap between Cooking Recipes and Dish Images with Latent Variable Model
• Probing Contextualized Sentence Representations with Visual Awareness
• Upgrading the Newsroom: An Automated Image Selection System for News Articles
• Target-Oriented Deformation of Visual-Semantic Embedding Space
• A Strong and Robust Baseline for Text-Image Matching
• Multiple Visual-Semantic Embedding for Video Retrieval from Query Sentence
• Retrieving and Highlighting Action with Spatiotemporal Reference