TAB-VCR: Tags and Attributes based Visual Commonsense Reasoning Baselines

Abstract

Reasoning is an important ability that we learn from a very early age. Yet, reasoning is extremely hard for algorithms. Despite impressive recent progress that has been reported on tasks that necessitate reasoning, such as visual question answering and visual dialog, models often exploit biases in datasets. To develop models with better reasoning abilities, recently, the new visual commonsense reasoning (VCR) task has been introduced. Not only do models have to answer questions, but also do they have to provide a reason for the given answer. The proposed baseline achieved compelling results, leveraging a meticulously designed model composed of LSTM modules and attention nets. Here we show that a much simpler model obtained by ablating and pruning the existing intricate baseline can perform better with half the number of trainable parameters. By associating visual features with attribute information and better text to image grounding, we obtain further improvements for our simpler & effective baseline, TAB-VCR. We show that this approach results in a 5.3%, 4.4% and 6.5% absolute improvement over the previous state-of-the-art on question answering, answer justification and holistic VCR.

arXiv commentAccepted to NeurIPS 2019. Project page: https://deanplayerljx.github.io/tabvcr

References in corpus (15)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• VisualBERT: A Simple and Performant Baseline for Vision and Language
• VQA: Visual Question Answering
• Exploring Models and Data for Image Question Answering
• ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering
• A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input
• Towards Automatic Learning of Procedures from Web Instructional Videos
• UNITER: UNiversal Image-TExt Representation Learning
• Hadamard Product for Low-rank Bilinear Pooling
• A Focused Dynamic Attention Model for Visual Question Answering
• Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
• Visual Madlibs: Fill in the blank Image Generation and Question Answering
• STAIR Actions: A Video Dataset of Everyday Home Actions
• TVQA+: Spatio-Temporal Grounding for Video Question Answering
• Neural Modular Control for Embodied Question Answering