Exploiting BERT For Multimodal Target Sentiment Classification Through Input Space Translation

Abstract

Multimodal target/aspect sentiment classification combines multimodal sentiment analysis and aspect/target sentiment classification. The goal of the task is to combine vision and language to understand the sentiment towards a target entity in a sentence. Twitter is an ideal setting for the task because it is inherently multimodal, highly emotional, and affects real world events. However, multimodal tweets are short and accompanied by complex, possibly irrelevant images. We introduce a two-stream model that translates images in input space using an object-aware transformer followed by a single-pass non-autoregressive text generation approach. We then leverage the translation to construct an auxiliary sentence that provides multimodal information to a language model. Our approach increases the amount of text available to the language model and distills the object-level information in complex images. We achieve state-of-the-art performance on two multimodal Twitter datasets without modifying the internals of the language model to accept multimodal data, demonstrating the effectiveness of our translation. In addition, we explain a failure mode of a popular approach for aspect sentiment analysis when applied to tweets. Our code is available at \textcolor{blue}{\url{https://github.com/codezakh/exploiting-BERT-thru-translation}}.

arXiv commentACM Multimedia 2021 Oral

References in corpus (8)

• Challenges in Deploying Machine Learning: a Survey of Case Studies
• MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos
• Pretrained Transformers as Universal Computation Engines
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• Exploiting BERT for End-to-End Aspect-based Sentiment Analysis
• Reproducibility, Replicability and Beyond: Assessing Production Readiness of Aspect Based Sentiment Analysis in the Wild
• Integrating Multimodal Information in Large Pretrained Transformers
• Transform and Tell: Entity-Aware News Image Captioning

Cited by in corpus (4)

• Counterfactual Reasoning for Out-of-distribution Multimodal Sentiment Analysis
• Few-shot Multimodal Sentiment Analysis based on Multimodal Probabilistic Fusion Prompts
• New Benchmark Dataset and Fine-Grained Cross-Modal Fusion Framework for Vietnamese Multimodal Aspect-Category Sentiment Analysis
• Robust Multimodal Sentiment Analysis of Image-Text Pairs by Distribution-Based Feature Recovery and Fusion