Active Learning for Visual Question Answering: An Empirical Study

Abstract

We present an empirical study of active learning for Visual Question Answering, where a deep VQA model selects informative question-image pairs from a pool and queries an oracle for answers to maximally improve its performance under a limited query budget. Drawing analogies from human learning, we explore cramming (entropy), curiosity-driven (expected model change), and goal-driven (expected error reduction) active learning approaches, and propose a fast and effective goal-driven active learning scoring function to pick question-image pairs for deep VQA models under the Bayesian Neural Network framework. We find that deep VQA models need large amounts of training data before they can start asking informative questions. But once they do, all three approaches outperform the random selection baseline and achieve significant query savings. For the scenario where the model is allowed to ask generic questions about images but is evaluated only on specific questions (e.g., questions whose answer is either yes or no), our proposed goal-driven scoring function performs the best.

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Hierarchical Question-Image Co-Attention for Visual Question Answering
• VQA: Visual Question Answering
• Active Learning for Convolutional Neural Networks: A Core-Set Approach
• Deep Bayesian Active Learning with Image Data
• Bayesian Active Learning for Classification and Preference Learning
• Systematic evaluation of CNN advances on the ImageNet
• Ask Your Neurons: A Neural-based Approach to Answering Questions about Images
• Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation
• Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
• End-to-end optimization of goal-driven and visually grounded dialogue systems

Cited by in corpus (9)

• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• Deep Ensemble Bayesian Active Learning : Addressing the Mode Collapse issue in Monte Carlo dropout via Ensembles
• Replay in Deep Learning: Current Approaches and Missing Biological Elements
• Deep Bayesian Active Learning for Multiple Correct Outputs
• Single-Modal Entropy based Active Learning for Visual Question Answering
• Being Negative but Constructively: Lessons Learnt from Creating Better Visual Question Answering Datasets
• Interactive Text Ranking with Bayesian Optimisation: A Case Study on Community QA and Summarisation
• Learning to Rank for Active Learning: A Listwise Approach
• Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering