Exploration on Grounded Word Embedding: Matching Words and Images with Image-Enhanced Skip-Gram Model
arXiv:1809.02765
Abstract
Word embedding is designed to represent the semantic meaning of a word with low dimensional vectors. The state-of-the-art methods of learning word embeddings (word2vec and GloVe) only use the word co-occurrence information. The learned embeddings are real number vectors, which are obscure to human. In this paper, we propose an Image-Enhanced Skip-Gram Model to learn grounded word embeddings by representing the word vectors in the same hyper-plane with image vectors. Experiments show that the image vectors and word embeddings learned by our model are highly correlated, which indicates that our model is able to provide a vivid image-based explanation to the word embeddings.
References in corpus (11)
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Deep Residual Learning for Image Recognition
- Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
- Word Representations via Gaussian Embedding
- Multimodal Convolutional Neural Networks for Matching Image and Sentence
- Label Embedding Network: Learning Label Representation for Soft Training of Deep Networks
- Multimodal Word Distributions
- Query and Output: Generating Words by Querying Distributed Word Representations for Paraphrase Generation
- Zero-Resource Neural Machine Translation with Multi-Agent Communication Game
- Regularizing Output Distribution of Abstractive Chinese Social Media Text Summarization for Improved Semantic Consistency
- simNet: Stepwise Image-Topic Merging Network for Generating Detailed and Comprehensive Image Captions