Equal But Not The Same: Understanding the Implicit Relationship Between Persuasive Images and Text

Abstract

Images and text in advertisements interact in complex, non-literal ways. The two channels are usually complementary, with each channel telling a different part of the story. Current approaches, such as image captioning methods, only examine literal, redundant relationships, where image and text show exactly the same content. To understand more complex relationships, we first collect a dataset of advertisement interpretations for whether the image and slogan in the same visual advertisement form a parallel (conveying the same message without literally saying the same thing) or non-parallel relationship, with the help of workers recruited on Amazon Mechanical Turk. We develop a variety of features that capture the creativity of images and the specificity or ambiguity of text, as well as methods that analyze the semantics within and across channels. We show that our method outperforms standard image-text alignment approaches on predicting the parallel/non-parallel relationship between image and text.

To appear in BMVC2018

References in corpus (7)

• NLTK: The Natural Language Toolkit
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• 6 Seconds of Sound and Vision: Creativity in Micro-Videos
• Speaking the Same Language: Matching Machine to Human Captions by Adversarial Training
• Context-aware Captions from Context-agnostic Supervision
• Self-supervised learning of visual features through embedding images into text topic spaces
• AMC: Attention guided Multi-modal Correlation Learning for Image Search

Cited by in corpus (8)

• The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
• Beyond Visual Semantics: Exploring the Role of Scene Text in Image Understanding
• Cap2Det: Learning to Amplify Weak Caption Supervision for Object Detection
• DM${}^2$S${}^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention
• Enhancing Cross-Modal Contextual Congruence for Crowdfunding Success using Knowledge-infused Learning
• Intentonomy: a Dataset and Study towards Human Intent Understanding
• QuTI! Quantifying Text-Image Consistency in Multimodal Documents
• Deep Multi-Modal Sets