Adapting Visual Question Answering Models for Enhancing Multimodal Community Q&A Platforms
arXiv:1808.09648
Abstract
Question categorization and expert retrieval methods have been crucial for information organization and accessibility in community question & answering (CQA) platforms. Research in this area, however, has dealt with only the text modality. With the increasing multimodal nature of web content, we focus on extending these methods for CQA questions accompanied by images. Specifically, we leverage the success of representation learning for text and images in the visual question answering (VQA) domain, and adapt the underlying concept and architecture for automated category classification and expert retrieval on image-based questions posted on Yahoo! Chiebukuro, the Japanese counterpart of Yahoo! Answers. To the best of our knowledge, this is the first work to tackle the multimodality challenge in CQA, and to adapt VQA models for tasks on a more ecologically valid source of visual questions. Our analysis of the differences between visual QA and community QA data drives our proposal of novel augmentations of an attention method tailored for CQA, and use of auxiliary tasks for learning better grounding features. Our final model markedly outperforms the text-only and VQA model baselines for both tasks of classification and expert retrieval on real-world multimodal CQA data.
arXiv commentSubmitted for review at CIKM 2019
References in corpus (12)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- An Overview of Multi-Task Learning in Deep Neural Networks
- Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
- VQA: Visual Question Answering
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
- Exploring Models and Data for Image Question Answering
- Simple Baseline for Visual Question Answering
- A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input
- Pythia v0.1: the Winning Entry to the VQA Challenge 2018
- Explicit Knowledge-based Reasoning for Visual Question Answering
- A Deep Learning Approach for Expert Identification in Question Answering Communities
- Soft Seeded SSL Graphs for Unsupervised Semantic Similarity-based Retrieval