Speech-Based Visual Question Answering

Abstract

This paper introduces speech-based visual question answering (VQA), the task of generating an answer given an image and a spoken question. Two methods are studied: an end-to-end, deep neural network that directly uses audio waveforms as input versus a pipelined approach that performs ASR (Automatic Speech Recognition) on the question, followed by text-based visual question answering. Furthermore, we investigate the robustness of both methods by injecting various levels of noise into the spoken question and find both methods to be tolerate noise at similar levels.

References in corpus (7)

• Hierarchical Question-Image Co-Attention for Visual Question Answering
• Recurrent Models of Visual Attention
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• SoundNet: Learning Sound Representations from Unlabeled Video
• Comparing Speech and Keyboard Text Entry for Short Messages in Two Languages on Touchscreen Phones
• Dual Attention Networks for Multimodal Reasoning and Matching

Cited by in corpus (4)

• Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech
• From Visual to Acoustic Question Answering
• NAAQA: A Neural Architecture for Acoustic Question Answering
• Non-monotonic Logical Reasoning Guiding Deep Learning for Explainable Visual Question Answering