Learning Visually-Grounded Semantics from Contrastive Adversarial Samples
arXiv:1806.10348
Abstract
We study the problem of grounding distributional representations of texts on the visual domain, namely visual-semantic embeddings (VSE for short). Begin with an insightful adversarial attack on VSE embeddings, we show the limitation of current frameworks and image-text datasets (e.g., MS-COCO) both quantitatively and qualitatively. The large gap between the number of possible constitutions of real-world semantics and the size of parallel data, to a large extent, restricts the model to establish the link between textual semantics and visual concepts. We alleviate this problem by augmenting the MS-COCO image captioning datasets with textual contrastive adversarial samples. These samples are synthesized using linguistic rules and the WordNet knowledge base. The construction procedure is both syntax- and semantics-aware. The samples enforce the model to ground learned embeddings to concrete concepts within the image. This simple but powerful technique brings a noticeable improvement over the baselines on a diverse set of downstream tasks, in addition to defending known-type adversarial attacks. We release the codes at https://github.com/ExplorerFreda/VSE-C.
arXiv commentTo Appear at COLING 2018
References in corpus (7)
- Efficient Estimation of Word Representations in Vector Space
- Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
- ReasoNet: Learning to Stop Reading in Machine Comprehension
- Natural Language Inference over Interaction Space
- Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
- Delving into adversarial attacks on deep policies
- Training and Evaluating Multimodal Word Embeddings with Large-scale Web Annotated Images
Cited by in corpus (8)
- Adversarial Attacks on Deep Learning Models in Natural Language Processing: A Survey
- Visually Grounded Neural Syntax Acquisition
- UniVSE: Robust Visual Semantic Embeddings via Structured Semantic Representations
- Substructure Substitution: Structured Data Augmentation for NLP
- HAL: Improved Text-Image Matching by Mitigating Visual Semantic Hubs
- Multi-Head Attention with Diversity for Learning Grounded Multilingual Multimodal Representations
- A Strong and Robust Baseline for Text-Image Matching
- Retrieving and Highlighting Action with Spatiotemporal Reference