Learning semantic sentence representations from visually grounded language without lexical knowledge
arXiv:1903.11393 · doi:10.1017/S1351324919000196
Abstract
Current approaches to learning semantic representations of sentences often use prior word-level knowledge. The current study aims to leverage visual information in order to capture sentence level semantics without the need for word embeddings. We use a multimodal sentence encoder trained on a corpus of images with matching text captions to produce visually grounded sentence embeddings. Deep Neural Networks are trained to map the two modalities to a common embedding space such that for an image the corresponding caption can be retrieved and vice versa. We show that our model achieves results comparable to the current state-of-the-art on two popular image-caption retrieval benchmark data sets: MSCOCO and Flickr8k. We evaluate the semantic content of the resulting sentence embeddings using the data from the Semantic Textual Similarity benchmark task and show that the multimodal embeddings correlate well with human semantic similarity judgements. The system achieves state-of-the-art results on several of these benchmarks, which shows that a system trained solely on multimodal data, without assuming any word representations, is able to capture sentence level semantics. Importantly, this result shows that we do not need prior knowledge of lexical level semantics in order to model sentence level semantics. These findings demonstrate the importance of visual information in semantics.
References in corpus (18)
- Adam: A Method for Stochastic Optimization
- Efficient Estimation of Word Representations in Vector Space
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- LSTM: A Search Space Odyssey
- Distributed Representations of Sentences and Documents
- Microsoft COCO Captions: Data Collection and Evaluation Server
- Skip-Thought Vectors
- SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation
- A large annotated corpus for learning natural language inference
- SentEval: An Evaluation Toolkit for Universal Sentence Representations
- Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
- Predicting Visual Features from Text for Image and Video Caption Retrieval
- Representations of language in a model of visually grounded speech signal
- Snapshot Ensembles: Train 1, get M for free
- Learning Semantic Similarity for Very Short Texts
- Multimodal Convolutional Neural Networks for Matching Image and Sentence
- Learning Semantic Textual Similarity from Conversations
- Recurrent Neural Network based Part-of-Speech Tagger for Code-Mixed Social Media Text