The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale

Abstract

We present Open Images V4, a dataset of 9.2M images with unified annotations for image classification, object detection and visual relationship detection. The images have a Creative Commons Attribution license that allows to share and adapt the material, and they have been collected from Flickr without a predefined list of class names or tags, leading to natural class statistics and avoiding an initial design bias. Open Images V4 offers large scale across several dimensions: 30.1M image-level labels for 19.8k concepts, 15.4M bounding boxes for 600 object classes, and 375k visual relationship annotations involving 57 classes. For object detection in particular, we provide 15x more bounding boxes than the next largest datasets (15.4M boxes on 1.9M images). The images often show complex scenes with several objects (8 annotated objects per image on average). We annotated visual relationships between them, which support visual relationship detection, an emerging task that requires structured reasoning. We provide in-depth comprehensive statistics about the dataset, we validate the quality of the annotations, we study how the performance of several modern models evolves with increasing amounts of training data, and we demonstrate two applications made possible by having unified annotations of multiple types coexisting in the same images. We hope that the scale, quality, and variety of Open Images V4 will foster further research and innovation even beyond the areas of image classification, object detection, and visual relationship detection.

Accepted to International Journal of Computer Vision, 2020

References in corpus (12)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• SSD: Single Shot MultiBox Detector
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distributed Representations of Words and Phrases and their Compositionality
• Distilling the Knowledge in a Neural Network
• Focal Loss for Dense Object Detection
• Visual Semantic Role Labeling
• Revisiting Unreasonable Effectiveness of Data in Deep Learning Era
• iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection
• Detecting Visual Relationships with Deep Relational Networks
• Visual Translation Embedding Network for Visual Relation Detection
• Deep Variation-structured Reinforcement Learning for Visual Relationship and Attribute Detection

Cited by in corpus (151)

• GOT-10k: A Large High-Diversity Benchmark for Generic Object Tracking in the Wild
• A Survey of Deep Learning-based Object Detection
• Object Detection in Aerial Images: A Large-Scale Benchmark and Challenges
• Weighted boxes fusion: Ensembling boxes from different object detection models
• DivideMix: Learning with Noisy Labels as Semi-supervised Learning
• The Emerging Trends of Multi-Label Learning
• FSS-1000: A 1000-Class Dataset for Few-Shot Segmentation
• A Primer on Motion Capture with Deep Learning: Principles, Pitfalls and Perspectives
• Deep Learning for Generic Object Detection: A Survey
• Real-Time Apple Detection System Using Embedded Systems With Hardware Accelerators: An Edge AI Application
• Scaling Out-of-Distribution Detection for Real-World Settings
• Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey
• Probabilistic two-stage detection
• Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biases
• RepPoints: Point Set Representation for Object Detection
• Robust Federated Learning with Noisy Labels
• Does Object Recognition Work for Everyone?
• A Deep Journey into Super-resolution: A survey
• Tencent ML-Images: A Large-Scale Multi-Label Image Database for Visual Representation Learning
• Occlusion Handling in Generic Object Detection: A Review
• Transferable Interactiveness Knowledge for Human-Object Interaction Detection
• Asymmetric Loss For Multi-Label Classification
• VinVL: Revisiting Visual Representations in Vision-Language Models
• PolarMask: Single Shot Instance Segmentation with Polar Representation
• Soft Sampling for Robust Object Detection
• The Best of Both Modes: Separately Leveraging RGB and Depth for Unseen Object Instance Segmentation
• MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without Tricks
• Large image datasets: A pyrrhic win for computer vision?
• D${}^2$-City: A Large-Scale Dashcam Video Dataset of Diverse Traffic Scenarios
• HAKE: Human Activity Knowledge Engine
• Sequence Level Semantics Aggregation for Video Object Detection
• A Robust Illumination-Invariant Camera System for Agricultural Applications
• Few-Shot Object Detection with Attention-RPN and Multi-Relation Detector
• Deep Learning based Switching Filter for Impulsive Noise Removal in Color Images
• Cross-Modal Contrastive Learning for Text-to-Image Generation
• An Analysis of Pre-Training on Object Detection
• E-LPIPS: Robust Perceptual Image Similarity via Random Transformation Ensembles
• Budgeted Training: Rethinking Deep Neural Network Training Under Resource Constraints
• Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrieval
• Equalization Loss for Long-Tailed Object Recognition
• BSN++: Complementary Boundary Regressor with Scale-Balanced Relation Modeling for Temporal Action Proposal Generation
• BREEDS: Benchmarks for Subpopulation Shift
• Empirical Upper Bound in Object Detection and More
• TAP: Text-Aware Pre-training for Text-VQA and Text-Caption
• WSOD^2: Learning Bottom-up and Top-down Objectness Distillation for Weakly-supervised Object Detection
• Learning a Deep ConvNet for Multi-label Classification with Partial Labels
• Comparison of State-of-the-Art Deep Learning APIs for Image Multi-Label Classification using Semantic Metrics
• Detailed 2D-3D Joint Representation for Human-Object Interaction
• Iterative Answer Prediction with Pointer-Augmented Multimodal Transformers for TextVQA
• Metamorphic Testing for Object Detection Systems
• Cross-dataset Training for Class Increasing Object Detection
• Dense RepPoints: Representing Visual Objects with Dense Point Sets
• When Does Self-supervision Improve Few-shot Learning?
• RetinaTrack: Online Single Stage Joint Detection and Tracking
• Guided Collaborative Training for Pixel-wise Semi-Supervised Learning
• A deep learning based solution for construction equipment detection: from development to deployment
• SimpleDet: A Simple and Versatile Distributed Framework for Object Detection and Instance Recognition
• Domain Adaptation Through Task Distillation
• 1st Place Solutions for OpenImage2019 -- Object Detection and Instance Segmentation
• MimicDet: Bridging the Gap Between One-Stage and Two-Stage Object Detection
• Optimizing JPEG Quantization for Classification Networks
• Webly Supervised Image Classification with Metadata: Automatic Noisy Label Correction via Visual-Semantic Graph
• A Survey on Deep Domain Adaptation and Tiny Object Detection Challenges, Techniques and Datasets
• Improving CNN-based Planar Object Detection with Geometric Prior Knowledge
• Segmenting Transparent Objects in the Wild
• FAIR1M: A Benchmark Dataset for Fine-grained Object Recognition in High-Resolution Remote Sensing Imagery
• Revisiting the Sibling Head in Object Detector
• Equalization Loss v2: A New Gradient Balance Approach for Long-tailed Object Detection
• GPS-Net: Graph Property Sensing Network for Scene Graph Generation
• Effective Label Propagation for Discriminative Semi-Supervised Domain Adaptation
• Oriented Boxes for Accurate Instance Segmentation
• Extraction of Complex DNN Models: Real Threat or Boogeyman?
• NOTE-RCNN: NOise Tolerant Ensemble RCNN for Semi-Supervised Object Detection
• Universal Lesion Detection by Learning from Multiple Heterogeneously Labeled Datasets
• Rethinking Natural Adversarial Examples for Classification Models
• Accelerating Deep Learning Inference via Learned Caches
• Self-supervised Single-view 3D Reconstruction via Semantic Consistency
• Evaluating Large-Vocabulary Object Detectors: The Devil is in the Details
• Boosting Weakly Supervised Object Detection with Progressive Knowledge Transfer
• Perception for Autonomous Systems (PAZ)
• Multi-Task Self-Training for Learning General Representations
• Data-Uncertainty Guided Multi-Phase Learning for Semi-Supervised Object Detection
• A Unified Framework for Generic, Query-Focused, Privacy Preserving and Update Summarization using Submodular Information Measures
• Learning Gaussian Maps for Dense Object Detection
• Learning from a Teacher using Unlabeled Data
• Neural Data Server: A Large-Scale Search Engine for Transfer Learning Data
• SoK: How Robust is Image Classification Deep Neural Network Watermarking? (Extended Version)
• SpatialSense: An Adversarially Crowdsourced Benchmark for Spatial Relation Recognition
• Detect-to-Retrieve: Efficient Regional Aggregation for Image Search
• The iMaterialist Fashion Attribute Dataset
• Learning Effective Visual Relationship Detector on 1 GPU
• Large-Scale Object Detection in the Wild from Imbalanced Multi-Labels
• 1st place solution for AVA-Kinetics Crossover in AcitivityNet Challenge 2020
• Context-Aware Image Inpainting with Learned Semantic Priors
• Light-weight Calibrator: a Separable Component for Unsupervised Domain Adaptation
• Learning an Efficient Network for Large-Scale Hierarchical Object Detection with Data Imbalance: 3rd Place Solution to Open Images Challenge 2019
• Object Detection with a Unified Label Space from Multiple Datasets
• Text-to-Image Generation Grounded by Fine-Grained User Attention
• Obj-GloVe: Scene-Based Contextual Object Embedding
• Structured Sparse R-CNN for Direct Scene Graph Generation
• 3D Spatial Recognition without Spatially Labeled 3D
• Homography Estimation with Convolutional Neural Networks Under Conditions of Variance
• Continual Learning: Tackling Catastrophic Forgetting in Deep Neural Networks with Replay Processes
• Egoshots, an ego-vision life-logging dataset and semantic fidelity metric to evaluate diversity in image captioning models
• Scaling Object Detection by Transferring Classification Weights
• Searching for Apparel Products from Images in the Wild
• Team PFDet's Methods for Open Images Challenge 2019
• DynaDog+T: A Parametric Animal Model for Synthetic Canine Image Generation
• Efficient Object Embedding for Spliced Image Retrieval
• What leads to generalization of object proposals?
• Natural Vocabulary Emerges from Free-Form Annotations
• DEEVA: A Deep Learning and IoT Based Computer Vision System to Address Safety and Security of Production Sites in Energy Industry
• TextTubes for Detecting Curved Text in the Wild
• Balance-Oriented Focal Loss with Linear Scheduling for Anchor Free Object Detection
• Contemplating real-world object classification
• Human De-occlusion: Invisible Perception and Recovery for Humans
• Language-guided Semantic Mapping and Mobile Manipulation in Partially Observable Environments
• CAPTION: Correction by Analyses, POS-Tagging and Interpretation of Objects using only Nouns
• Unsupervised Object Detection with LiDAR Clues
• Adapting Convolutional Neural Networks for Geographical Domain Shift
• Pseudo-Labeling for Small Lesion Detection on Diabetic Retinopathy Images
• REGRAD: A Large-Scale Relational Grasp Dataset for Safe and Object-Specific Robotic Grasping in Clutter
• LGA-RCNN: Loss-Guided Attention for Object Detection
• Natural Adversarial Objects
• Compositional Embeddings for Multi-Label One-Shot Learning
• eProduct: A Million-Scale Visual Search Benchmark to Address Product Recognition Challenges
• Dissecting Image Crops
• Efficient video annotation with visual interpolation and frame selection guidance
• High-Level Perceptual Similarity is Enabled by Learning Diverse Tasks
• Target Reaching Behaviour for Unfreezing the Robot in a Semi-Static and Crowded Environment
• Equalization Loss for Large Vocabulary Instance Segmentation
• Unified Signal Compression Using Generative Adversarial Networks
• SpotPatch: Parameter-Efficient Transfer Learning for Mobile Object Detection
• Beyond Categorical Label Representations for Image Classification
• Empirical Upper Bound, Error Diagnosis and Invariance Analysis of Modern Object Detectors
• Learned Image Compression for Machine Perception
• Privacy-Aware Activity Classification from First Person Office Videos
• Robust Assessment of Real-World Adversarial Examples
• Complementary Boundary Generator with Scale-Invariant Relation Modeling for Temporal Action Localization: Submission to ActivityNet Challenge 2020
• Model Composition: Can Multiple Neural Networks Be Combined into a Single Network Using Only Unlabeled Data?
• Semi-supervised Learning for Dense Object Detection in Retail Scenes
• Weakly Supervised Visual Semantic Parsing
• It's About Time: Analog Clock Reading in the Wild
• Knowledge Guided Learning: Towards Open Domain Egocentric Action Recognition with Zero Supervision
• CC-Loss: Channel Correlation Loss For Image Classification
• PANDA: A Gigapixel-level Human-centric Video Dataset
• USB: Universal-Scale Object Detection Benchmark
• Informative Object Annotations: Tell Me Something I Don't Know
• Augmented Bi-path Network for Few-shot Learning
• Shelf-Supervised Mesh Prediction in the Wild
• LVIS: A Dataset for Large Vocabulary Instance Segmentation