Understanding Visual Ads by Aligning Symbols and Objects using Co-Attention
arXiv:1807.01448
Abstract
We tackle the problem of understanding visual ads where given an ad image, our goal is to rank appropriate human generated statements describing the purpose of the ad. This problem is generally addressed by jointly embedding images and candidate statements to establish correspondence. Decoding a visual ad requires inference of both semantic and symbolic nuances referenced in an image and prior methods may fail to capture such associations especially with weakly annotated symbols. In order to create better embeddings, we leverage an attention mechanism to associate image proposals with symbols and thus effectively aggregate information from aligned multimodal representations. We propose a multihop co-attention mechanism that iteratively refines the attention map to ensure accurate attention estimation. Our attention based embedding model is learned end-to-end guided by a max-margin loss function. We show that our model outperforms other baselines on the benchmark Ad dataset and also show qualitative results to highlight the advantages of using multihop co-attention.
arXiv commentAccepted at CVPR 2018 workshop- Towards Automatic Understanding of Visual Advertisements
References in corpus (9)
- Adam: A Method for Stochastic Optimization
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
- Hierarchical Question-Image Co-Attention for Visual Question Answering
- The Cityscapes Dataset for Semantic Urban Scene Understanding
- Action Recognition using Visual Attention
- Attentional Pooling for Action Recognition
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
- Dual Attention Networks for Multimodal Reasoning and Matching