Pythia v0.1: the Winning Entry to the VQA Challenge 2018

Abstract

This document describes Pythia v0.1, the winning entry from Facebook AI Research (FAIR)'s A-STAR team to the VQA Challenge 2018. Our starting point is a modular re-implementation of the bottom-up top-down (up-down) model. We demonstrate that by making subtle but important changes to the model architecture and the learning rate schedule, fine-tuning image features, and adding data augmentation, we can significantly improve the performance of the up-down model on VQA v2.0 dataset -- from 65.67% to 70.22%. Furthermore, by using a diverse ensemble of models trained with different features and on different datasets, we are able to significantly improve over the 'standard' way of ensembling (i.e. same model with different random seeds) by 1.31%. Overall, we achieve 72.27% on the test-std split of the VQA v2.0 dataset. Our code in its entirety (training, evaluation, data-augmentation, ensembling) and pre-trained models are publicly available at: https://github.com/facebookresearch/pythia

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• VQA: Visual Question Answering
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Aggregated Residual Transformations for Deep Neural Networks

Cited by in corpus (48)

• VisualBERT: A Simple and Performant Baseline for Vision and Language
• ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
• LXMERT: Learning Cross-Modality Encoder Representations from Transformers
• Visual Entailment: A Novel Task for Fine-Grained Image Understanding
• How Much Can CLIP Benefit Vision-and-Language Tasks?
• Unified Vision-Language Pre-Training for Image Captioning and VQA
• Relation-Aware Graph Attention Network for Visual Question Answering
• LPF: A Language-Prior Feedback Objective Function for De-biased Visual Question Answering
• Vilio: State-of-the-art Visio-Linguistic Models applied to Hateful Memes
• Toward 3D Spatial Reasoning for Human-like Text-based Visual Question Answering
• Exploring Sparse Spatial Relation in Graph Inference for Text-Based VQA
• A Closer Look at the Robustness of Vision-and-Language Pre-trained Models
• MUREL: Multimodal Relational Reasoning for Visual Question Answering
• Systematic Generalization: What Is Required and Can It Be Learned?
• Human-Adversarial Visual Question Answering
• Multimodal Integration of Human-Like Attention in Visual Question Answering
• TAP: Text-Aware Pre-training for Text-VQA and Text-Caption
• In Defense of Grid Features for Visual Question Answering
• VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answering
• MoVie: Revisiting Modulated Convolutions for Visual Counting and Beyond
• Kaleido-BERT: Vision-Language Pre-training on Fashion Domain
• VQA-LOL: Visual Question Answering under the Lens of Logic
• Learning from Lexical Perturbations for Consistent Visual Question Answering
• Towards Escaping from Language Bias and OCR Error: Semantics-Centered Text Visual Question Answering
• OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge
• A Study on Multimodal and Interactive Explanations for Visual Question Answering
• All-in-One Image-Grounded Conversational Agents
• Dual Recurrent Attention Units for Visual Question Answering
• A Picture May Be Worth a Hundred Words for Visual Question Answering
• Visual Question Answering with Prior Class Semantics
• Generating Rationales in Visual Question Answering
• Point and Ask: Incorporating Pointing into Visual Question Answering
• Rephrasing visual questions by specifying the entropy of the answer distribution
• An Entropy Clustering Approach for Assessing Visual Question Difficulty
• Grounded Video Description
• Cross-Modal Generative Augmentation for Visual Question Answering
• Localize, Group, and Select: Boosting Text-VQA by Scene Text Modeling
• Multiple interaction learning with question-type prior knowledge for constraining answer search space in visual question answering
• Attention that does not Explain Away
• PyTorchPipe: a framework for rapid prototyping of pipelines combining language and vision
• Are VQA Systems RAD? Measuring Robustness to Augmented Data with Focused Interventions
• Adapting Visual Question Answering Models for Enhancing Multimodal Community Q&A Platforms
• Graph Relation Transformer: Incorporating pairwise object features into the Transformer architecture
• Learning Better Visual Dialog Agents with Pretrained Visual-Linguistic Representation
• NarrationBot and InfoBot: A Hybrid System for Automated Video Description
• Decoupled Box Proposal and Featurization with Ultrafine-Grained Semantic Labels Improve Image Captioning and Visual Question Answering
• Visual Question Answering Using Semantic Information from Image Descriptions
• Generating and Evaluating Explanations of Attended and Error-Inducing Input Regions for VQA Models