Compact Trilinear Interaction for Visual Question Answering
arXiv:1909.11874
Abstract
In Visual Question Answering (VQA), answers have a great correlation with question meaning and visual contents. Thus, to selectively utilize image, question and answer information, we propose a novel trilinear interaction model which simultaneously learns high level associations between these three inputs. In addition, to overcome the interaction complexity, we introduce a multimodal tensor-based PARALIND decomposition which efficiently parameterizes trilinear interaction between the three inputs. Moreover, knowledge distillation is first time applied in Free-form Opened-ended VQA. It is not only for reducing the computational cost and required memory but also for transferring knowledge from trilinear interaction model to bilinear interaction model. The extensive experiments on benchmarking datasets TDIUC, VQA-2.0, and Visual7W show that the proposed compact trilinear interaction model achieves state-of-the-art results when using a single model on all three datasets.
Accepted in ICCV 2019
References in corpus (11)
- Distilling the Knowledge in a Neural Network
- Do Deep Nets Really Need to be Deep?
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
- Simple Baseline for Visual Question Answering
- Multimodal Residual Learning for Visual QA
- Hadamard Product for Low-rank Bilinear Pooling
- A Focused Dynamic Attention Model for Visual Question Answering
- Training Recurrent Answering Units with Joint Loss Minimization for VQA
- Zero-Shot Visual Question Answering
- Towards a Visual Turing Challenge
- VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentation
Cited by in corpus (5)
- Knowledge Distillation: A Survey
- Improving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signals
- MRI-based Alzheimer's disease prediction via distilling the knowledge in multi-modal data
- Stroke Constrained Attention Network for Online Handwritten Mathematical Expression Recognition
- MOI-Mixer: Improving MLP-Mixer with Multi Order Interactions in Sequential Recommendation