VQA: Visual Question Answering

Abstract

We propose the task of free-form and open-ended Visual Question Answering (VQA). Given an image and a natural language question about the image, the task is to provide an accurate natural language answer. Mirroring real-world scenarios, such as helping the visually impaired, both the questions and answers are open-ended. Visual questions selectively target different areas of an image, including background details and underlying context. As a result, a system that succeeds at VQA typically needs a more detailed understanding of the image and complex reasoning than a system producing generic image captions. Moreover, VQA is amenable to automatic evaluation, since many open-ended answers contain only a few words or a closed set of answers that can be provided in a multiple-choice format. We provide a dataset containing ~0.25M images, ~0.76M questions, and ~10M answers (www.visualqa.org), and discuss the information it provides. Numerous baselines and methods for VQA are provided and compared with human performance. Our VQA demo is available on CloudCV (http://cloudcv.org/vqa).

The first three authors contributed equally. International Conference on Computer Vision (ICCV) 2015

References in corpus (10)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
• Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
• Show and Tell: A Neural Image Caption Generator
• Deep Visual-Semantic Alignments for Generating Image Descriptions
• CIDEr: Consensus-based Image Description Evaluation
• Yin and Yang: Balancing and Answering Binary Visual Questions
• Don't Just Listen, Use Your Imagination: Leveraging Visual Common Sense for Non-Visual Tasks

Cited by in corpus (482)

• Transformers in Vision: A Survey
• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• The Open Catalyst 2020 (OC20) Dataset and Community Challenges
• Dynamic Memory Networks for Visual and Textual Question Answering
• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Modulating early visual processing by language
• Simple Baseline for Visual Question Answering
• Scaling Egocentric Vision: The EPIC-KITCHENS Dataset
• Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers
• ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering
• Visual Question Answering: Datasets, Algorithms, and Future Challenges
• Adversarial Attacks on Deep Learning Models in Natural Language Processing: A Survey
• Pathologies of Neural Models Make Interpretations Difficult
• Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answering
• LXMERT: Learning Cross-Modality Encoder Representations from Transformers
• Cognitive Science in the era of Artificial Intelligence: A roadmap for reverse-engineering the infant language-learner
• BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
• WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning
• Learning to Compose and Reason with Language Tree Structures for Visual Grounding
• Natural Language Processing Advancements By Deep Learning: A Survey
• UNITER: UNiversal Image-TExt Representation Learning
• Ground Truth Evaluation of Neural Network Explanations with CLEVR-XAI
• Supervised Multimodal Bitransformers for Classifying Images and Text
• Visual Entailment: A Novel Task for Fine-Grained Image Understanding
• Pythia v0.1: the Winning Entry to the VQA Challenge 2018
• QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension
• Context-Aware Visual Policy Network for Fine-Grained Image Captioning
• ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data
• Multimodal Machine Learning: A Survey and Taxonomy
• How Much Can CLIP Benefit Vision-and-Language Tasks?
• Deep Identity-aware Transfer of Facial Attributes
• Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense
• ImageNet pre-trained models with batch normalization
• Learning Dual Semantic Relations with Graph Attention for Image-Text Matching
• Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation
• A Focused Dynamic Attention Model for Visual Question Answering
• Generation and Comprehension of Unambiguous Object Descriptions
• Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints
• Context-Aware Visual Policy Network for Sequence-Level Image Captioning
• Vision-and-Dialog Navigation
• ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph
• Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training
• Cross-modal Knowledge Reasoning for Knowledge-based Visual Question Answering
• Learning to Reason: End-to-End Module Networks for Visual Question Answering
• Ask, Attend and Answer: Exploring Question-Guided Spatial Attention for Visual Question Answering
• Video Question Answering via Attribute-Augmented Attention Network Learning
• Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering
• Explicit Knowledge-based Reasoning for Visual Question Answering
• Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks
• Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
• Learning Cooperative Visual Dialog Agents with Deep Reinforcement Learning
• Visual Reference Resolution using Attention Memory for Visual Dialog
• Deep Image Retrieval: Learning global representations for image search
• Regularizing Deep Neural Networks by Noise: Its Interpretation and Optimization
• Focal Visual-Text Attention for Visual Question Answering
• Learning Language-Visual Embedding for Movie Understanding with Natural-Language
• Learning to Generalize from Sparse and Underspecified Rewards
• Relation Networks for Object Detection
• Image Captioning for Effective Use of Language Models in Knowledge-Based Visual Question Answering
• Image as Data: Automated Visual Content Analysis for Political Science
• Adversarial NLI: A New Benchmark for Natural Language Understanding
• Recurrent Neural Networks for Semantic Instance Segmentation
• Person Search with Natural Language Description
• Out of the Box: Reasoning with Graph Convolution Nets for Factual Visual Question Answering
• Learning like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images
• Visual Translation Embedding Network for Visual Relation Detection
• Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches
• Relation-Aware Graph Attention Network for Visual Question Answering
• Learning Deep Structure-Preserving Image-Text Embeddings
• Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
• Context-Dependent Diffusion Network for Visual Relationship Detection
• Dataset Condensation with Differentiable Siamese Augmentation
• GLAC Net: GLocal Attention Cascading Networks for Multi-image Cued Story Generation
• Deep Variation-structured Reinforcement Learning for Visual Relationship and Attribute Detection
• Natural Language Understanding with Distributed Representation
• LPF: A Language-Prior Feedback Objective Function for De-biased Visual Question Answering
• CLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialog
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• TGIF: A New Dataset and Benchmark on Animated GIF Description
• Convolutional Network for Attribute-driven and Identity-preserving Human Face Generation
• A Multimodal Framework for the Detection of Hateful Memes
• Ask Me Anything: Free-form Visual Question Answering Based on Knowledge from External Sources
• Building a Large-scale Multimodal Knowledge Base System for Answering Visual Queries
• Dual Attention Networks for Multimodal Reasoning and Matching
• CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning
• ShapeWorld - A new test methodology for multimodal language understanding
• Visual7W: Grounded Question Answering in Images
• Visual Question Answering: A Survey of Methods and Datasets
• Uncovering Temporal Context for Video Question and Answering
• Towards Ecologically Valid Research on Language User Interfaces
• Action2Vec: A Crossmodal Embedding Approach to Action Learning
• Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions
• Incorporating External Knowledge to Answer Open-Domain Visual Questions with Dynamic Memory Networks
• Visual Question Answering for Cultural Heritage
• Image Question Answering using Convolutional Neural Network with Dynamic Parameter Prediction
• SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning
• Unshuffling Data for Improved Generalization
• VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research
• Yin and Yang: Balancing and Answering Binary Visual Questions
• Compositional Memory for Visual Question Answering
• Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation
• Unifying Multimodal Transformer for Bi-directional Image and Text Generation
• TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering
• Identity-Aware Textual-Visual Matching with Latent Co-attention
• CLOSURE: Assessing Systematic Generalization of CLEVR Models
• A Multimodal Memes Classification: A Survey and Open Research Issues
• Emergence of Compositional Language with Deep Generational Transmission
• Multimodal Unified Attention Networks for Vision-and-Language Interactions
• Variational Context: Exploiting Visual and Textual Context for Grounding Referring Expressions
• What value do explicit high level concepts have in vision to language problems?
• Measuring and Improving Consistency in Pretrained Language Models
• Creativity: Generating Diverse Questions using Variational Autoencoders
• End-to-end optimization of goal-driven and visually grounded dialogue systems
• Learning to Compose Dynamic Tree Structures for Visual Contexts
• RAVEN: A Dataset for Relational and Analogical Visual rEasoNing
• It Takes Two to Tango: Towards Theory of AI's Mind
• ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
• Revisiting Visual Question Answering Baselines
• Are we pretraining it right? Digging deeper into visio-linguistic pretraining
• Show, Adapt and Tell: Adversarial Training of Cross-domain Image Captioner
• Scene Graph Generation from Objects, Phrases and Region Captions
• Don't Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases
• PAM: Understanding Product Images in Cross Product Category Attribute Extraction
• Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog
• ST-HOI: A Spatial-Temporal Baseline for Human-Object Interaction Detection in Videos
• Visual Question Generation as Dual Task of Visual Question Answering
• Fashion IQ: A New Dataset Towards Retrieving Images by Natural Language Feedback
• What Makes Training Multi-Modal Classification Networks Hard?
• Towards a Robust Deep Neural Network in Texts: A Survey
• DVQA: Understanding Data Visualizations via Question Answering
• Structured Attentions for Visual Question Answering
• Scene Graph Reasoning with Prior Visual Relationship for Visual Question Answering
• VQS: Linking Segmentations to Questions and Answers for Supervised Attention in VQA and Question-Focused Semantic Segmentation
• TVQA+: Spatio-Temporal Grounding for Video Question Answering
• Audio Visual Scene-Aware Dialog (AVSD) Challenge at DSTC7
• Context-aware Captions from Context-agnostic Supervision
• MemexQA: Visual Memex Question Answering
• Natural Language Guided Visual Relationship Detection
• Neural Language Generation: Formulation, Methods, and Evaluation
• Motion-Appearance Co-Memory Networks for Video Question Answering
• Look Before You Leap: Bridging Model-Free and Model-Based Reinforcement Learning for Planned-Ahead Vision-and-Language Navigation
• Situation Recognition with Graph Neural Networks
• Survey of Visual Question Answering: Datasets and Techniques
• An Analysis of Visual Question Answering Algorithms
• Question-Guided Hybrid Convolution for Visual Question Answering
• Guided Feature Transformation (GFT): A Neural Language Grounding Module for Embodied Agents
• Task-driven Visual Saliency and Attention-based Visual Question Answering
• Understanding Early Word Learning in Situated Artificial Agents
• Document Collection Visual Question Answering
• Understanding the Role of Scene Graphs in Visual Question Answering
• REMIND Your Neural Network to Prevent Catastrophic Forgetting
• Recommending Themes for Ad Creative Design via Visual-Linguistic Representations
• Question Answering is a Format; When is it Useful?
• Weakly-supervised Visual Grounding of Phrases with Linguistic Structures
• ViTAA: Visual-Textual Attributes Alignment in Person Search by Natural Language
• Learning Two-Branch Neural Networks for Image-Text Matching Tasks
• Combating Human Trafficking with Deep Multimodal Models
• SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels
• Language Tasks and Language Games: On Methodology in Current Natural Language Processing Research
• Attacking Visual Language Grounding with Adversarial Examples: A Case Study on Neural Image Captioning
• Adversarial Reinforced Instruction Attacker for Robust Vision-Language Navigation
• GuessWhat?! Visual object discovery through multi-modal dialogue
• CraftAssist: A Framework for Dialogue-enabled Interactive Agents
• Multimodal Attribute Extraction
• Active Learning for Visual Question Answering: An Empirical Study
• From FiLM to Video: Multi-turn Question Answering with Multi-modal Context
• Contrastive Visual-Linguistic Pretraining
• Recent Advances and Trends in Multimodal Deep Learning: A Review
• OpenViDial: A Large-Scale, Open-Domain Dialogue Dataset with Visual Contexts
• Entropy-Enhanced Multimodal Attention Model for Scene-Aware Dialogue Generation
• Stories for Images-in-Sequence by using Visual and Narrative Components
• Asking the Difficult Questions: Goal-Oriented Visual Question Generation via Intermediate Rewards
• Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answering
• Consensus-Aware Visual-Semantic Embedding for Image-Text Matching
• InterpNET: Neural Introspection for Interpretable Deep Learning
• Shallow2Deep: Indoor Scene Modeling by Single Image Understanding
• Learning Convolutional Text Representations for Visual Question Answering
• Learning to Globally Edit Images with Textual Description
• Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks
• Recent Advances in Neural Program Synthesis
• VQA-E: Explaining, Elaborating, and Enhancing Your Answers for Visual Questions
• Structured Multimodal Attentions for TextVQA
• Are You Talking to Me? Reasoned Visual Dialog Generation through Adversarial Learning
• Saliency-Guided Attention Network for Image-Sentence Matching
• Grounding Referring Expressions in Images by Variational Context
• DualNet: Domain-Invariant Network for Visual Question Answering
• ChartNet: Visual Reasoning over Statistical Charts using MAC-Networks
• Visual Question Answering with Memory-Augmented Networks
• VQABQ: Visual Question Answering by Basic Questions
• Robustness Analysis of Visual QA Models by Basic Questions
• Video Fill in the Blank with Merging LSTMs
• MoVie: Revisiting Modulated Convolutions for Visual Counting and Beyond
• FlipDial: A Generative Model for Two-Way Visual Dialogue
• Pre-Trained Models: Past, Present and Future
• Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models
• Linguistically-aware Attention for Reducing the Semantic-Gap in Vision-Language Tasks
• Multimodal Hierarchical Reinforcement Learning Policy for Task-Oriented Visual Dialog
• Speech-Based Visual Question Answering
• Object Relation Detection Based on One-shot Learning
• Visual Entailment Task for Visually-Grounded Language Learning
• Factor Graph Attention
• A Diagram Is Worth A Dozen Images
• Visual Relationship Detection using Scene Graphs: A Survey
• A Restricted Visual Turing Test for Deep Scene and Event Understanding
• Tell-and-Answer: Towards Explainable Visual Question Answering using Attributes and Captions
• CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication
• Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructions
• End-to-end Learning of Deep Visual Representations for Image Retrieval
• Visual Text Correction
• Unanswerable Questions about Images and Texts
• Dual Attention Networks for Visual Reference Resolution in Visual Dialog
• Making History Matter: History-Advantage Sequence Training for Visual Dialog
• Joint Image Captioning and Question Answering
• LEAF-QA: Locate, Encode & Attend for Figure Question Answering
• Retinal Vessel Segmentation under Extreme Low Annotation: A Generative Adversarial Network Approach
• Explaining the Unexplained: A CLass-Enhanced Attentive Response (CLEAR) Approach to Understanding Deep Neural Networks
• Image-Question-Answer Synergistic Network for Visual Dialog
• Sentiment Analysis of Fashion Related Posts in Social Media
• Question-Conditioned Counterfactual Image Generation for VQA
• ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answering
• MCQA: Multimodal Co-attention Based Network for Question Answering
• Automatic Generation of Grounded Visual Questions
• Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigation
• Scene Graph Reasoning for Visual Question Answering
• Supervised and Unsupervised Transfer Learning for Question Answering
• Sitatapatra: Blocking the Transfer of Adversarial Samples
• MMFT-BERT: Multimodal Fusion Transformer with BERT Encodings for Visual Question Answering
• SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answering
• Going Beneath the Surface: Evaluating Image Captioning for Grammaticality, Truthfulness and Diversity
• Grounded Agreement Games: Emphasizing Conversational Grounding in Visual Dialogue Settings
• Answering Visual What-If Questions: From Actions to Predicted Scene Descriptions
• An Empirical Evaluation of Visual Question Answering for Novel Objects
• The Color of the Cat is Gray: 1 Million Full-Sentences Visual Question Answering (FSVQA)
• Automatic Description Generation from Images: A Survey of Models, Datasets, and Evaluation Measures
• RecipeQA: A Challenge Dataset for Multimodal Comprehension of Cooking Recipes
• The Visual QA Devil in the Details: The Impact of Early Fusion and Batch Norm on CLEVR
• Deep Multimodal Neural Architecture Search
• VQA-LOL: Visual Question Answering under the Lens of Logic
• Segmentations-Leak: Membership Inference Attacks and Defenses in Semantic Image Segmentation
• Matching Images and Text with Multi-modal Tensor Fusion and Re-ranking
• Compositional Generalization for Primitive Substitutions
• Learning Visual Features from Large Weakly Supervised Data
• Context, Attention and Audio Feature Explorations for Audio Visual Scene-Aware Dialog
• Language bias in Visual Question Answering: A Survey and Taxonomy
• Fooling Vision and Language Models Despite Localization and Attention Mechanism
• Learning Visual Question Answering by Bootstrapping Hard Attention
• Leveraging Medical Visual Question Answering with Supporting Facts
• VQD: Visual Query Detection in Natural Scenes
• Information Maximizing Visual Question Generation
• A Survey on Machine Reading Comprehension: Tasks, Evaluation Metrics and Benchmark Datasets
• Survey of Recent Advances in Visual Question Answering
• Video Highlight Prediction Using Audience Chat Reactions
• Do Explanations make VQA Models more Predictable to a Human?
• Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision
• Dense Captioning with Joint Inference and Visual Context
• A Study on Multimodal and Interactive Explanations for Visual Question Answering
• Faithful Multimodal Explanation for Visual Question Answering
• A Qualitative Comparison of CoQA, SQuAD 2.0 and QuAC
• Question Type Guided Attention in Visual Question Answering
• Improving Deep Visual Representation for Person Re-identification by Global and Local Image-language Association
• An Interpretable Model for Scene Graph Generation
• E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning
• Multi-Modal Generative Adversarial Network for Short Product Title Generation in Mobile E-Commerce
• Textually Enriched Neural Module Networks for Visual Question Answering
• MarioQA: Answering Questions by Watching Gameplay Videos
• On transfer learning using a MAC model variant
• Deep Binaries: Encoding Semantic-Rich Cues for Efficient Textual-Visual Cross Retrieval
• Figure Captioning with Reasoning and Sequence-Level Training
• Leveraging Visual Question Answering to Improve Text-to-Image Synthesis
• Multi-task Learning for Universal Sentence Embeddings: A Thorough Evaluation using Transfer and Auxiliary Tasks
• Understanding Image and Text Simultaneously: a Dual Vision-Language Machine Comprehension Task
• Scene Graph Parsing as Dependency Parsing
• LiveBot: Generating Live Video Comments Based on Visual and Textual Contexts
• Grid-VLP: Revisiting Grid Features for Vision-Language Pre-training
• A dataset and exploration of models for understanding video data through fill-in-the-blank question-answering
• Improving Visual Question Answering by Referring to Generated Paragraph Captions
• Loss re-scaling VQA: Revisiting the LanguagePrior Problem from a Class-imbalance View
• Answer Them All! Toward Universal Visual Question Answering Models
• Dual Recurrent Attention Units for Visual Question Answering
• Conditional Text Generation for Harmonious Human-Machine Interaction
• 2nd Place Solution to the GQA Challenge 2019
• Generating Question Relevant Captions to Aid Visual Question Answering
• Visual Referring Expression Recognition: What Do Systems Actually Learn?
• Image Classification for Arabic: Assessing the Accuracy of Direct English to Arabic Translations
• Visual Question Answering on 360° Images
• A System for Automated Image Editing from Natural Language Commands
• Detecting depression in dyadic conversations with multimodal narratives and visualizations
• Multi-level Multimodal Common Semantic Space for Image-Phrase Grounding
• Self-supervised pre-training and contrastive representation learning for multiple-choice video QA
• A Joint Speaker-Listener-Reinforcer Model for Referring Expressions
• Concept-Centric Visual Turing Tests for Method Validation
• TallyQA: Answering Complex Counting Questions
• Multimodal Dialogue State Tracking By QA Approach with Data Augmentation
• Bidirectional Beam Search: Forward-Backward Inference in Neural Sequence Models for Fill-in-the-Blank Image Captioning
• Recursive Visual Attention in Visual Dialog
• The ACRV Picking Benchmark (APB): A Robotic Shelf Picking Benchmark to Foster Reproducible Research
• Multimodal Learning for Hateful Memes Detection
• Break It Down: A Question Understanding Benchmark
• Commonly Uncommon: Semantic Sparsity in Situation Recognition
• Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioning
• Expressing Visual Relationships via Language
• Investigating Biases in Textual Entailment Datasets
• Visual Grounding Methods for VQA are Working for the Wrong Reasons!
• C3VQG: Category Consistent Cyclic Visual Question Generation
• Probing Emergent Semantics in Predictive Agents via Question Answering
• Modality-Balanced Models for Visual Dialogue
• Transferable Representation Learning in Vision-and-Language Navigation
• Weakly supervised cross-domain alignment with optimal transport
• Graph Edit Distance Reward: Learning to Edit Scene Graph
• The Probabilistic Object Detection Challenge
• Keep it Consistent: Topic-Aware Storytelling from an Image Stream via Iterative Multi-agent Communication
• TutorialVQA: Question Answering Dataset for Tutorial Videos
• Self-Guiding Multimodal LSTM - when we do not have a perfect training dataset for image captioning
• Answering Questions about Data Visualizations using Efficient Bimodal Fusion
• Visual Query Answering by Entity-Attribute Graph Matching and Reasoning
• UniVSE: Robust Visual Semantic Embeddings via Structured Semantic Representations
• Interactive Language Acquisition with One-shot Visual Concept Learning through a Conversational Game
• From Visual to Acoustic Question Answering
• Towards a Better Metric for Evaluating Question Generation Systems
• ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integration
• Being Negative but Constructively: Lessons Learnt from Creating Better Visual Question Answering Datasets
• Deep Multi-Modal Image Correspondence Learning
• Seeing is Knowing! Fact-based Visual Question Answering using Knowledge Graph Embeddings
• Attentive Explanations: Justifying Decisions and Pointing to the Evidence (Extended Abstract)
• Convolutional Neural Networks for Aerial Multi-Label Pedestrian Detection
• Can Transformers Reason About Effects of Actions?
• I Want This Product but Different : Multimodal Retrieval with Synthetic Query Expansion
• KANDINSKYPatterns -- An experimental exploration environment for Pattern Analysis and Machine Intelligence
• A Picture May Be Worth a Hundred Words for Visual Question Answering
• Multimodal Differential Network for Visual Question Generation
• Visual Fashion-Product Search at SK Planet
• Learning to Disambiguate by Asking Discriminative Questions
• TAB-VCR: Tags and Attributes based Visual Commonsense Reasoning Baselines
• Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization
• VidLanKD: Improving Language Understanding via Video-Distilled Knowledge Transfer
• NAAQA: A Neural Architecture for Acoustic Question Answering
• Conversational AI Systems for Social Good: Opportunities and Challenges
• Grounding Open-Domain Instructions to Automate Web Support Tasks
• What is More Likely to Happen Next? Video-and-Language Future Event Prediction
• Multi-label classification of promotions in digital leaflets using textual and visual information
• Learning to Represent Image and Text with Denotation Graph
• ORD: Object Relationship Discovery for Visual Dialogue Generation
• Revisiting Deep Architectures for Head Motion Prediction in 360° Videos
• Multi-Layer Content Interaction Through Quaternion Product For Visual Question Answering
• MTLE: A Multitask Learning Encoder of Visual Feature Representations for Video and Movie Description
• Multimodal Sentiment Analysis with Word-Level Fusion and Reinforcement Learning
• MMED: A Multi-domain and Multi-modality Event Dataset
• Quantifying and Alleviating the Language Prior Problem in Visual Question Answering
• Spatial Memory for Context Reasoning in Object Detection
• ViP-CNN: Visual Phrase Guided Convolutional Neural Network
• Object-Level Context Modeling For Scene Classification with Context-CNN
• Answering Image Riddles using Vision and Reasoning through Probabilistic Soft Logic
• MULE: Multimodal Universal Language Embedding
• Kite: Automatic speech recognition for unmanned aerial vehicles
• Question Relevance in Visual Question Answering
• Cross-Dataset Adaptation for Visual Question Answering
• The VQA-Machine: Learning How to Use Existing Vision Algorithms to Answer New Questions
• ManyModalQA: Modality Disambiguation and QA over Diverse Inputs
• Lessons learned in multilingual grounded language learning
• Visual Curiosity: Learning to Ask Questions to Learn Visual Recognition
• Reasoning about Fine-grained Attribute Phrases using Reference Games
• Large Graph Exploration via Subgraph Discovery and Decomposition
• Deep Sparse Coding for Invariant Multimodal Halle Berry Neurons
• AiR: Attention with Reasoning Capability
• Modularized Textual Grounding for Counterfactual Resilience
• Learning Models for Actions and Person-Object Interactions with Transfer to Question Answering
• Cross-Modal Generative Augmentation for Visual Question Answering
• Variational Inference for Learning Representations of Natural Language Edits
• Mining Deep And-Or Object Structures via Cost-Sensitive Question-Answer-Based Active Annotations
• Understanding Art through Multi-Modal Retrieval in Paintings
• Modeling Image Virality with Pairwise Spatial Transformer Networks
• Which Ads to Show? Advertisement Image Assessment with Auxiliary Information via Multi-step Modality Fusion
• Think Visually: Question Answering through Virtual Imagery
• Question Guided Modular Routing Networks for Visual Question Answering
• Trying Bilinear Pooling in Video-QA
• Check It Again: Progressive Visual Question Answering via Visual Entailment
• Learning to Caption Images through a Lifetime by Asking Questions
• A Better Loss for Visual-Textual Grounding
• CAPTION: Correction by Analyses, POS-Tagging and Interpretation of Objects using only Nouns
• Attend What You Need: Motion-Appearance Synergistic Networks for Video Question Answering
• Evaluating Text-to-Image Matching using Binary Image Selection (BISON)
• MELINDA: A Multimodal Dataset for Biomedical Experiment Method Classification
• Query-controllable Video Summarization
• Video Question Generation via Cross-Modal Self-Attention Networks Learning
• Leveraging Topics and Audio Features with Multimodal Attention for Audio Visual Scene-Aware Dialog
• Pairwise Relations Discriminator for Unsupervised Raven's Progressive Matrices
• Open Challenges on Generating Referring Expressions for Human-Robot Interaction
• Paying Attention to Descriptions Generated by Image Captioning Models
• Visual Question Answering as Reading Comprehension
• MemeFaceGenerator: Adversarial Synthesis of Chinese Meme-face from Natural Sentences
• Benchmark Visual Question Answer Models by using Focus Map
• Using Depth for Improving Referring Expression Comprehension in Real-World Environments
• DVD: A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialogue
• What is needed for simple spatial language capabilities in VQA?
• PyTorchPipe: a framework for rapid prototyping of pipelines combining language and vision
• The Amazing Mysteries of the Gutter: Drawing Inferences Between Panels in Comic Book Narratives
• Enforcing Reasoning in Visual Commonsense Reasoning
• Improved RAMEN: Towards Domain Generalization for Visual Question Answering
• A First Look: Towards Explainable TextVQA Models via Visual and Textual Explanations
• Designing Multimodal Datasets for NLP Challenges
• X-modaler: A Versatile and High-performance Codebase for Cross-modal Analytics
• Multi-Head Attention with Diversity for Learning Grounded Multilingual Multimodal Representations
• Weakly Supervised Image Annotation and Segmentation with Objects and Attributes
• Leveraging Visual Question Answering for Image-Caption Ranking
• Human-centric Relation Segmentation: Dataset and Solution
• What data do we need for training an AV motion planner?
• Abductive Reasoning as Self-Supervision for Common Sense Question Answering
• Are VQA Systems RAD? Measuring Robustness to Augmented Data with Focused Interventions
• Visual Reasoning with Natural Language
• Multiple interaction learning with question-type prior knowledge for constraining answer search space in visual question answering
• DeepSIC: Deep Semantic Image Compression
• Exploring Context, Attention and Audio Features for Audio Visual Scene-Aware Dialog
• Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning
• Cascaded Mutual Modulation for Visual Reasoning
• The Wisdom of MaSSeS: Majority, Subjectivity, and Semantic Similarity in the Evaluation of VQA
• Understand, Compose and Respond - Answering Visual Questions by a Composition of Abstract Procedures
• Weak Supervision helps Emergence of Word-Object Alignment and improves Vision-Language Tasks
• CompGuessWhat?!: A Multi-task Evaluation Framework for Grounded Language Learning
• Auto-Classification of Retinal Diseases in the Limit of Sparse Data Using a Two-Streams Machine Learning Model
• Interactive Text2Pickup Network for Natural Language based Human-Robot Collaboration
• Cross-modal Speaker Verification and Recognition: A Multilingual Perspective
• LearningWord Embeddings for Low-resource Languages by PU Learning
• Zero-shot task adaptation by homoiconic meta-mapping
• HUMBO: Bridging Response Generation and Facial Expression Synthesis
• BiST: Bi-directional Spatio-Temporal Reasoning for Video-Grounded Dialogues
• On Architectures for Including Visual Information in Neural Language Models for Image Description
• Multimodal Logical Inference System for Visual-Textual Entailment
• Aligned Image-Word Representations Improve Inductive Transfer Across Vision-Language Tasks
• Holistic Multi-modal Memory Network for Movie Question Answering
• Neuro-Symbolic VQA: A review from the perspective of AGI desiderata
• Convolutional Neural Networks for Aerial Vehicle Detection and Recognition
• Semantic bottleneck for computer vision tasks
• Adapting Visual Question Answering Models for Enhancing Multimodal Community Q&A Platforms
• On the Flip Side: Identifying Counterexamples in Visual Question Answering
• Towards Understanding End-of-trip Instructions in a Taxi Ride Scenario
• Effect of Various Regularizers on Model Complexities of Neural Networks in Presence of Input Noise
• Modal-aware Features for Multimodal Hashing
• Open-Ended Visual Question Answering by Multi-Modal Domain Adaptation
• Customized Image Narrative Generation via Interactive Visual Question Generation and Answering
• Modulated Self-attention Convolutional Network for VQA
• Assisting human experts in the interpretation of their visual process: A case study on assessing copper surface adhesive potency
• Challenging Images For Minds and Machines
• Decoupled Box Proposal and Featurization with Ultrafine-Grained Semantic Labels Improve Image Captioning and Visual Question Answering
• Describing Unseen Videos via Multi-Modal Cooperative Dialog Agents
• Unsupervised Keyword Extraction for Full-sentence VQA
• A Free Lunch in Generating Datasets: Building a VQG and VQA System with Attention and Humans in the Loop
• Using Thought-Provoking Children's Questions to Drive Artificial Intelligence Research
• Applying recent advances in Visual Question Answering to Record Linkage
• Evaluating the Representational Hub of Language and Vision Models
• Visual Question Answering Using Semantic Information from Image Descriptions
• XAlgo: a Design Probe of Explaining Algorithms' Internal States via Question-Answering
• Cross-Modality Relevance for Reasoning on Language and Vision
• We went to look for meaning and all we got were these lousy representations: aspects of meaning representation for computational semantics
• Graph-based Heuristic Search for Module Selection Procedure in Neural Module Network
• Interpretable Neural Computation for Real-World Compositional Visual Question Answering
• OOWL500: Overcoming Dataset Collection Bias in the Wild
• Pay Attention to Those Sets! Learning Quantification from Images
• A Revised Generative Evaluation of Visual Dialogue
• Deep Unified Multimodal Embeddings for Understanding both Content and Users in Social Media Networks
• Towards Instance Segmentation with Object Priority: Prominent Object Detection and Recognition
• Recurrent Models for Situation Recognition
• SIRI: Spatial Relation Induced Network For Spatial Description Resolution
• Analysis on Image Set Visual Question Answering
• What is Multimodality?
• 'Just because you are right, doesn't mean I am wrong': Overcoming a Bottleneck in the Development and Evaluation of Open-Ended Visual Question Answering (VQA) Tasks
• RL-CSDia: Representation Learning of Computer Science Diagrams
• VGNMN: Video-grounded Neural Module Network to Video-Grounded Language Tasks
• Comparing Visual Reasoning in Humans and AI
• Inspiring Computer Vision System Solutions
• VideoMCC: a New Benchmark for Video Comprehension
• WebQA: Multihop and Multimodal QA
• ICDAR 2021 Competition on Document VisualQuestion Answering
• CARLS: Cross-platform Asynchronous Representation Learning System
• GEM: A General Evaluation Benchmark for Multimodal Tasks
• Building a Video-and-Language Dataset with Human Actions for Multimodal Logical Inference
• X-GGM: Graph Generative Modeling for Out-of-Distribution Generalization in Visual Question Answering
• Learning Articulated Motion Models from Visual and Lingual Signals
• Reasoning Over History: Context Aware Visual Dialog
• Proposing Plausible Answers for Open-ended Visual Question Answering
• Middle-Out Decoding
• Composing Text and Image for Image Retrieval - An Empirical Odyssey