Universal Multimodal Representation for Language Understanding

Abstract

Representation learning is the foundation of natural language processing (NLP). This work presents new methods to employ visual information as assistant signals to general NLP tasks. For each sentence, we first retrieve a flexible number of images either from a light topic-image lookup table extracted over the existing sentence-image pairs or a shared cross-modal embedding space that is pre-trained on out-of-shelf text-image pairs. Then, the text and images are encoded by a Transformer encoder and convolutional neural network, respectively. The two sequences of representations are further fused by an attention layer for the interaction of the two modalities. In this study, the retrieval process is controllable and flexible. The universal visual representation overcomes the lack of large-scale bilingual sentence-image pairs. Our method can be easily applied to text-only tasks without manually annotated multimodal parallel corpora. We apply the proposed method to a wide range of natural language generation and understanding tasks, including neural machine translation, natural language inference, and semantic similarity. Experimental results show that our method is generally effective for different tasks and languages. Analysis indicates that the visual signals enrich textual representations of content words, provide fine-grained grounding information about the relationship between concepts and events, and potentially conduce to disambiguation.

arXiv commentAccepted by IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)

References in corpus (17)

• Adam: A Method for Stochastic Optimization
• Neural Machine Translation by Jointly Learning to Align and Translate
• FaceNet: A Unified Embedding for Face Recognition and Clustering
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• VL-BERT: Pre-training of Generic Visual-Linguistic Representations
• SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation
• Data Augmentation for Low-Resource Neural Machine Translation
• Phrase-Based & Neural Unsupervised Machine Translation
• Learning to Compose and Reason with Language Tree Structures for Visual Grounding
• Glyce: Glyph-vectors for Chinese Character Representations
• Finding beans in burgers: Deep semantic-visual embedding with localization
• Regularizing Deep Neural Networks by Noise: Its Interpretation and Optimization
• Visually Grounded Neural Syntax Acquisition
• Dynamic Context-guided Capsule Network for Multimodal Machine Translation
• An Exploration of Word Embedding Initialization in Deep-Learning Tasks
• Revisiting Image-Language Networks for Open-ended Phrase Detection
• Good for Misconceived Reasons: An Empirical Revisiting on the Need for Visual Context in Multimodal Machine Translation

Cited by in corpus (3)

• From Rule-Based Models to Deep Learning Transformers Architectures for Natural Language Processing and Sign Language Translation Systems: Survey, Taxonomy and Performance Evaluation
• Natural Language Processing with Commonsense Knowledge: A Survey
• A Multimodal-Multitask Framework with Cross-modal Relation and Hierarchical Interactive Attention for Semantic Comprehension