Empirical Upper Bound, Error Diagnosis and Invariance Analysis of Modern Object Detectors

Abstract

Object detection remains as one of the most notorious open problems in computer vision. Despite large strides in accuracy in recent years, modern object detectors have started to saturate on popular benchmarks raising the question of how far we can reach with deep learning tools and tricks. Here, by employing 2 state-of-the-art object detection benchmarks, and analyzing more than 15 models over 4 large scale datasets, we I) carefully determine the upper bound in AP, which is 91.6% on VOC (test2007), 78.2% on COCO (val2017), and 58.9% on OpenImages V4 (validation), regardless of the IOU threshold. These numbers are much better than the mAP of the best model (47.9% on VOC, and 46.9% on COCO; IOUs=.5:.05:.95), II) characterize the sources of errors in object detectors, in a novel and intuitive way, and find that classification error (confusion with other classes and misses) explains the largest fraction of errors and weighs more than localization and duplicate errors, and III) analyze the invariance properties of models when surrounding context of an object is removed, when an object is placed in an incongruent background, and when images are blurred or flipped vertically. We find that models generate a lot of boxes on empty regions and that context is more important for detecting small objects than larger ones. Our work taps into the tight relationship between object detection and object recognition and offers insights for building better models. Our code is publicly available at https://github.com/aliborji/Deetctionupper bound.git.

arXiv commentarXiv admin note: substantial text overlap with arXiv:1911.12451

References in corpus (17)

• Explaining and Harnessing Adversarial Examples
• YOLOv3: An Incremental Improvement
• FCOS: Fully Convolutional One-Stage Object Detection
• MMDetection: Open MMLab Detection Toolbox and Benchmark
• Object Detection in 20 Years: A Survey
• The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale
• Robustness May Be at Odds with Accuracy
• Systematic evaluation of CNN advances on the ImageNet
• Deep Learning for Generic Object Detection: A Survey
• Why do deep convolutional networks generalize so poorly to small image transformations?
• Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming
• Physical Adversarial Examples for Object Detectors
• TensorMask: A Foundation for Dense Object Segmentation
• An Analysis of Pre-Training on Object Detection
• Empirical Upper Bound in Object Detection and More
• APRICOT: A Dataset of Physical Adversarial Attacks on Object Detection
• Object Recognition with and without Objects