Training and Evaluating Multimodal Word Embeddings with Large-scale Web Annotated Images

Abstract

In this paper, we focus on training and evaluating effective word embeddings with both text and visual information. More specifically, we introduce a large-scale dataset with 300 million sentences describing over 40 million images crawled and downloaded from publicly available Pins (i.e. an image with sentence descriptions uploaded by users) on Pinterest. This dataset is more than 200 times larger than MS COCO, the standard large-scale image dataset with sentence descriptions. In addition, we construct an evaluation dataset to directly assess the effectiveness of word embeddings in terms of finding semantically similar or related words and phrases. The word/phrase pairs in this evaluation dataset are collected from the click data with millions of users in an image search system, thus contain rich semantic relationships. Based on these datasets, we propose and compare several Recurrent Neural Networks (RNNs) based multimodal (text and image) models. Experiments show that our model benefits from incorporating the visual information into the word embeddings, and a weight sharing strategy is crucial for learning such multimodal embeddings. The project page is: http://www.stat.ucla.edu/~junhua.mao/multimodal_embedding.html

Appears in NIPS 2016. The datasets introduced in this work will be gradually released on the project page

Cited by in corpus (9)

• Self-supervised transfer learning of physiological representations from free-living wearable data
• Learning Visually-Grounded Semantics from Contrastive Adversarial Samples
• Data-driven models and computational tools for neurolinguistics: a language technology perspective
• Evolution of a Web-Scale Near Duplicate Image Detection System
• Evaluating Multimodal Representations on Sentence Similarity: vSTS, Visual Semantic Textual Similarity Dataset
• Deriving Visual Semantics from Spatial Context: An Adaptation of LSA and Word2Vec to generate Object and Scene Embeddings from Images
• Harmonized Multimodal Learning with Gaussian Process Latent Variable Models
• Goal-driven text descriptions for images
• Learning Zero-Shot Multifaceted Visually Grounded Word Embeddings via Multi-Task Training