Probing Contextualized Sentence Representations with Visual Awareness
arXiv:1911.02971
Abstract
We present a universal framework to model contextualized sentence representations with visual awareness that is motivated to overcome the shortcomings of the multimodal parallel data with manual annotations. For each sentence, we first retrieve a diversity of images from a shared cross-modal embedding space, which is pre-trained on a large-scale of text-image pairs. Then, the texts and images are respectively encoded by transformer encoder and convolutional neural network. The two sequences of representations are further fused by a simple and effective attention layer. The architecture can be easily applied to text-only natural language processing tasks without manually annotating multimodal parallel corpora. We apply the proposed method on three tasks, including neural machine translation, natural language inference and sequence labeling and experimental results verify the effectiveness.
References in corpus (15)
- Adam: A Method for Stochastic Optimization
- Distributed Representations of Words and Phrases and their Compositionality
- VL-BERT: Pre-training of Generic Visual-Linguistic Representations
- LXMERT: Learning Cross-Modality Encoder Representations from Transformers
- Multi-Task Deep Neural Networks for Natural Language Understanding
- VideoBERT: A Joint Model for Video and Language Representation Learning
- Glyce: Glyph-vectors for Chinese Character Representations
- How multilingual is Multilingual BERT?
- Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
- Finding beans in burgers: Deep semantic-visual embedding with localization
- Visually Grounded Neural Syntax Acquisition
- Semantic Sentence Matching with Densely-connected Recurrent and Co-attentive Information
- SG-Net: Syntax-Guided Machine Reading Comprehension
- Semantics-aware BERT for Language Understanding
- Distilling Translations with Visual Awareness