Integrating Multimodal Information in Large Pretrained Transformers

Abstract

Recent Transformer-based contextual word representations, including BERT and XLNet, have shown state-of-the-art performance in multiple disciplines within NLP. Fine-tuning the trained contextual models on task-specific datasets has been the key to achieving superior performance downstream. While fine-tuning these pre-trained models is straightforward for lexical applications (applications with only language modality), it is not trivial for multimodal language (a growing area in NLP focused on modeling face-to-face communication). Pre-trained models don't have the necessary components to accept two extra modalities of vision and acoustic. In this paper, we proposed an attachment to BERT and XLNet called Multimodal Adaptation Gate (MAG). MAG allows BERT and XLNet to accept multimodal nonverbal data during fine-tuning. It does so by generating a shift to internal representation of BERT and XLNet; a shift that is conditioned on the visual and acoustic modalities. In our experiments, we study the commonly used CMU-MOSI and CMU-MOSEI datasets for multimodal sentiment analysis. Fine-tuning MAG-BERT and MAG-XLNet significantly boosts the sentiment analysis performance over previous baselines as well as language-only fine-tuning of BERT and XLNet. On the CMU-MOSI dataset, MAG-XLNet achieves human-level multimodal sentiment analysis performance for the first time in the NLP community.

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Distributed Representations of Words and Phrases and their Compositionality
• MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos
• Learning Factorized Multimodal Representations
• Visualizing and Measuring the Geometry of BERT
• VideoBERT: A Joint Model for Video and Language Representation Learning
• Efficient Low-rank Multimodal Fusion with Modality-Specific Factors
• Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors

Cited by in corpus (8)

• Exploiting BERT For Multimodal Target Sentiment Classification Through Input Space Translation
• Span-based Localizing Network for Natural Language Video Localization
• Cross-media Structured Common Space for Multimedia Event Extraction
• DocStruct: A Multimodal Method to Extract Hierarchy Structure in Document for General Form Understanding
• Complaint Identification in Social Media with Transformer Networks
• Audio-Visual Transformer Based Crowd Counting
• Modeling the Severity of Complaints in Social Media
• Multimodal Embeddings from Language Models