Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
arXiv:1602.07332
Abstract
Despite progress in perceptual tasks such as image classification, computers still perform poorly on cognitive tasks such as image description and question answering. Cognition is core to tasks that involve not just recognizing, but reasoning about our visual world. However, models used to tackle the rich content in images for cognitive tasks are still being trained using the same datasets designed for perceptual tasks. To achieve success at cognitive tasks, models need to understand the interactions and relationships between objects in an image. When asked "What vehicle is the person riding?", computers will need to identify the objects in an image as well as the relationships riding(man, carriage) and pulling(horse, carriage) in order to answer correctly that "the person is riding a horse-drawn carriage". In this paper, we present the Visual Genome dataset to enable the modeling of such relationships. We collect dense annotations of objects, attributes, and relationships within each image to learn these models. Specifically, our dataset contains over 100K images where each image has an average of 21 objects, 18 attributes, and 18 pairwise relationships between objects. We canonicalize the objects, attributes, relationships, and noun phrases in region descriptions and questions answer pairs to WordNet synsets. Together, these annotations represent the densest and largest dataset of image descriptions, objects, attributes, relationships, and question answers.
44 pages, 37 figures
Cited by in corpus (103)
- Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
- Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
- Florence: A New Foundation Model for Computer Vision
- Visual Question Answering: Datasets, Algorithms, and Future Challenges
- Learning a Text-Video Embedding from Incomplete and Heterogeneous Data
- Visual Entailment: A Novel Task for Fine-Grained Image Understanding
- ImageNet pre-trained models with batch normalization
- Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts
- Recurrent Topic-Transition GAN for Visual Paragraph Generation
- Transferable Interactiveness Knowledge for Human-Object Interaction Detection
- See, Hear, and Read: Deep Aligned Representations
- Visual Relationship Detection with Internal and External Linguistic Knowledge Distillation
- Relation-Aware Graph Attention Network for Visual Question Answering
- Context-Aware Embeddings for Automatic Art Analysis
- Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
- Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling
- Visual7W: Grounded Question Answering in Images
- Visual Question Answering for Cultural Heritage
- SCA-CNN: Spatial and Channel-wise Attention in Convolutional Networks for Image Captioning
- Long-Term Feature Banks for Detailed Video Understanding
- Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog
- DVQA: Understanding Data Visualizations via Question Answering
- Scene Graph Reasoning with Prior Visual Relationship for Visual Question Answering
- Unpaired Image Captioning via Scene Graph Alignments
- SPEECH-COCO: 600k Visually Grounded Spoken Captions Aligned to MSCOCO Data Set
- The More You Know: Using Knowledge Graphs for Image Classification
- Learning the Best Pooling Strategy for Visual Semantic Embedding
- An Analysis of Visual Question Answering Algorithms
- Situation Recognition with Graph Neural Networks
- General Multi-label Image Classification with Transformers
- Weakly-supervised Visual Grounding of Phrases with Linguistic Structures
- Using Scene Graph Context to Improve Image Generation
- PPR-FCN: Weakly Supervised Visual Relation Detection via Parallel Pairwise R-FCN
- Weakly-supervised learning of visual relations
- Counterfactual Critic Multi-Agent Training for Scene Graph Generation
- Baidu-UTS Submission to the EPIC-Kitchens Action Recognition Challenge 2019
- In Defense of Grid Features for Visual Question Answering
- Improving Image Captioning by Leveraging Intra- and Inter-layer Global Representation in Transformer Network
- Revisiting Image-Language Networks for Open-ended Phrase Detection
- Image Scene Graph Generation (SGG) Benchmark
- SEIGAN: Towards Compositional Image Generation by Simultaneously Learning to Segment, Enhance, and Inpaint
- Scaling Up Vision-Language Pre-training for Image Captioning
- MoVie: Revisiting Modulated Convolutions for Visual Counting and Beyond
- Grounding Spatio-Semantic Referring Expressions for Human-Robot Interaction
- Learning to Segment Every Thing
- Behind the Scene: Revealing the Secrets of Pre-trained Vision-and-Language Models
- Learning Visual Knowledge Memory Networks for Visual Question Answering
- CoDraw: Collaborative Drawing as a Testbed for Grounded Goal-driven Communication
- More Grounded Image Captioning by Distilling Image-Text Matching Model
- VrR-VG: Refocusing Visually-Relevant Relationships
- OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge
- Matching Images and Text with Multi-modal Tensor Fusion and Re-ranking
- AutoCaption: Image Captioning with Neural Architecture Search
- Semantic Amodal Segmentation
- Machine Learning with World Knowledge: The Position and Survey
- VIOLIN: A Large-Scale Dataset for Video-and-Language Inference
- An Interpretable Model for Scene Graph Generation
- Improving Weakly Supervised Visual Grounding by Contrastive Knowledge Distillation
- Question Type Guided Attention in Visual Question Answering
- Scene Graph Parsing by Attention Graph
- Video Highlight Prediction Using Audience Chat Reactions
- Fine-grained Recognition in the Wild: A Multi-Task Domain Adaptation Approach
- Black-box Testing of First-Order Logic Ontologies Using WordNet
- Understanding Image and Text Simultaneously: a Dual Vision-Language Machine Comprehension Task
- Image Generation from Layout
- Dual-Glance Model for Deciphering Social Relationships
- AUTO-DISCERN: Autonomous Driving Using Common Sense Reasoning
- Tag Prediction at Flickr: a View from the Darkroom
- Lexically-constrained Text Generation through Commonsense Knowledge Extraction and Injection
- Unpaired Cross-lingual Image Caption Generation with Self-Supervised Rewards
- Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models
- Human-like Controllable Image Captioning with Verb-specific Semantic Roles
- Assessing Data Quality of Annotations with Krippendorff Alpha For Applications in Computer Vision
- Answering Questions about Data Visualizations using Efficient Bimodal Fusion
- SMP Challenge: An Overview of Social Media Prediction Challenge 2019
- Joint Visual Semantic Reasoning: Multi-Stage Decoder for Text Recognition
- Towards Accurate Text-based Image Captioning with Content Diversity Exploration
- Edited Media Understanding Frames: Reasoning About the Intent and Implications of Visual Misinformation
- Point and Ask: Incorporating Pointing into Visual Question Answering
- What and Where: A Context-based Recommendation System for Object Insertion
- Real-time Visual Object Tracking with Natural Language Description
- Triplet-Aware Scene Graph Embeddings
- ACD: Action Concept Discovery from Image-Sentence Corpora
- Spatial Memory for Context Reasoning in Object Detection
- Sampled Image Tagging and Retrieval Methods on User Generated Content
- Modeling Explicit Concerning States for Reinforcement Learning in Visual Dialogue
- Trying Bilinear Pooling in Video-QA
- CAPTION: Correction by Analyses, POS-Tagging and Interpretation of Objects using only Nouns
- Knowledge-Enriched Visual Storytelling
- Evaluating Text-to-Image Matching using Binary Image Selection (BISON)
- Reliable Label Bootstrapping for Semi-Supervised Learning
- Understanding Spatial Relations through Multiple Modalities
- The Amazing Mysteries of the Gutter: Drawing Inferences Between Panels in Comic Book Narratives
- Learning Action Concept Trees and Semantic Alignment Networks from Image-Description Data
- Generating Holistic 3D Scene Abstractions for Text-based Image Retrieval
- Sharing Cognition: Human Gesture and Natural Language Grounding Based Planning and Navigation for Indoor Robots
- Learning Single/Multi-Attribute of Object with Symmetry and Group
- Better Understanding Hierarchical Visual Relationship for Image Caption
- Constructing a Visual Relationship Authenticity Dataset
- Affinity Graph Supervision for Visual Recognition
- Daily Turking: Designing Longitudinal Daily-task Studies on Mechanical Turk
- ParNet: Position-aware Aggregated Relation Network for Image-Text matching
- The Focus-Aspect-Polarity Model for Predicting Subjective Noun Attributes in Images