Empirical Upper Bound in Object Detection and More

Abstract

Object detection remains as one of the most notorious open problems in computer vision. Despite large strides in accuracy in recent years, modern object detectors have started to saturate on popular benchmarks raising the question of how far we can reach with deep learning tools and tricks. Here, by employing 2 state-of-the-art object detection benchmarks, and analyzing more than 15 models over 4 large scale datasets, we I) carefully determine the upperbound in AP, which is 91.6% on VOC (test2007), 78.2% on COCO (val2017), and 58.9% on OpenImages V4 (validation), regardless of the IOU. These numbers are much better than the mAP of the best model1 (47.9% on VOC, and 46.9% on COCO; IOUs=.5:.95), II) characterize the sources of errors in object detectors, in a novel and intuitive way, and find that classification error (confusion with other classes and misses) explains the largest fraction of errors and weighs more than localization and duplicate errors, and III) analyze the invariance properties of models when surrounding context of an object is removed, when an object is placed in an incongruent background, and when images are blurred or flipped vertically. We find that models generate boxes on empty regions and that context is more important for detecting small objects than larger ones. Our work taps into the tight relationship between recognition and detection and offers insights for building better models.

References in corpus (12)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• FCOS: Fully Convolutional One-Stage Object Detection
• MMDetection: Open MMLab Detection Toolbox and Benchmark
• Object Detection in 20 Years: A Survey
• EfficientDet: Scalable and Efficient Object Detection
• The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale
• Do ImageNet Classifiers Generalize to ImageNet?
• Systematic evaluation of CNN advances on the ImageNet
• Why do deep convolutional networks generalize so poorly to small image transformations?
• Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming
• An Empirical Study of Spatial Attention Mechanisms in Deep Networks
• An Analysis of Pre-Training on Object Detection

Cited by in corpus (5)

• A Free Lunch for Unsupervised Domain Adaptive Object Detection without Source Data
• ObjectNet Dataset: Reanalysis and Correction
• Scale-aware Automatic Augmentation for Object Detection
• Box Re-Ranking: Unsupervised False Positive Suppression for Domain Adaptive Pedestrian Detection
• Empirical Upper Bound, Error Diagnosis and Invariance Analysis of Modern Object Detectors