Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming

Abstract

The ability to detect objects regardless of image distortions or weather conditions is crucial for real-world applications of deep learning like autonomous driving. We here provide an easy-to-use benchmark to assess how object detection models perform when image quality degrades. The three resulting benchmark datasets, termed Pascal-C, Coco-C and Cityscapes-C, contain a large variety of image corruptions. We show that a range of standard object detection models suffer a severe performance loss on corrupted images (down to 30--60\% of the original performance). However, a simple data augmentation trick---stylizing the training images---leads to a substantial increase in robustness across corruption type, severity and dataset. We envision our comprehensive benchmark to track future progress towards building robust object detection models. Benchmark, code and data are publicly available.

21 pages, 10 figures, 1 dragon

References in corpus (18)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Intriguing properties of neural networks
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Confounding variables can degrade generalization performance of radiological deep learning models
• Semantic Foggy Scene Understanding with Synthetic Data
• Focal Loss for Dense Object Detection
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• MMDetection: Open MMLab Detection Toolbox and Benchmark
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• Generalisation in humans and deep neural networks
• Deformable ConvNets v2: More Deformable, Better Results
• Why do deep convolutional networks generalize so poorly to small image transformations?
• Examining the Impact of Blur on Recognition by Convolutional Networks
• UA-DETRAC: A New Benchmark and Protocol for Multi-Object Detection and Tracking
• Rain Removal in Traffic Surveillance: Does it Matter?
• Towards Practical Verification of Machine Learning: The Case of Computer Vision Systems
• D${}^2$-City: A Large-Scale Dashcam Video Dataset of Diverse Traffic Scenarios
• All Weather Perception: Joint Data Association, Tracking, and Classification for Autonomous Ground Vehicles

Cited by in corpus (44)

• Shortcut Learning in Deep Neural Networks
• Deep learning tools for the measurement of animal behavior in neuroscience
• A Primer on Motion Capture with Deep Learning: Principles, Pitfalls and Perspectives
• Improving robustness against common corruptions by covariate shift adaptation
• Generalize then Adapt: Source-Free Domain Adaptive Semantic Segmentation
• Partial success in closing the gap between human and machine vision
• Test-Time Adaptation to Distribution Shift by Confidence Maximization and Input Transformation
• Learning perturbation sets for robust machine learning
• Lost in Pruning: The Effects of Pruning Neural Networks beyond Test Accuracy
• Computing Systems for Autonomous Driving: State-of-the-Art and Challenges
• Increasing the Robustness of Semantic Segmentation Models with Painting-by-Numbers
• Empirical Upper Bound in Object Detection and More
• Stochastic-YOLO: Efficient Probabilistic Object Detection under Dataset Shifts
• Does enhanced shape bias improve neural network robustness to common corruptions?
• Proposal Learning for Semi-Supervised Object Detection
• Towards robust vision by multi-task learning on monkey visual cortex
• Evaluating Robustness to Context-Sensitive Feature Perturbations of Different Granularities
• Bayesian Few-Shot Classification with One-vs-Each Pólya-Gamma Augmented Gaussian Processes
• Benchmarking the Robustness of Instance Segmentation Models
• Robustness testing of AI systems: A case study for traffic sign recognition
• Closer Look at the Uncertainty Estimation in Semantic Segmentation under Distributional Shift
• When Human Pose Estimation Meets Robustness: Adversarial Algorithms and Benchmarks
• Visual Representation Learning Does Not Generalize Strongly Within the Same Domain
• Robust Semantic Segmentation with Superpixel-Mix
• MmWave Radar and Vision Fusion for Object Detection in Autonomous Driving: A Review
• Revisiting Knowledge Distillation for Object Detection
• Gradient Shaping: Enhancing Backdoor Attack Against Reverse Engineering
• Robust and Accurate Object Detection via Adversarial Learning
• Robustness in Compressed Neural Networks for Object Detection
• Shift Equivariance in Object Detection
• SWAD: Domain Generalization by Seeking Flat Minima
• Rainy screens: Collecting rainy datasets, indoors
• Source-Free Adaptation to Measurement Shift via Bottom-Up Feature Restoration
• StyleAugment: Learning Texture De-biased Representations by Style Augmentation without Pre-defined Textures
• Utilizing Network Properties to Detect Erroneous Inputs
• UNO: Uncertainty-aware Noisy-Or Multimodal Fusion for Unanticipated Input Degradation
• On Exploring and Improving Robustness of Scene Text Detection Models
• Post-Hoc Domain Adaptation via Guided Data Homogenization
• Retaining Image Feature Matching Performance Under Low Light Conditions
• Disentangled Variational Information Bottleneck for Multiview Representation Learning
• Empirical Upper Bound, Error Diagnosis and Invariance Analysis of Modern Object Detectors
• Auto-Split: A General Framework of Collaborative Edge-Cloud AI
• Comparing the Machine Readability of Traffic Sign Pictograms in Austria and Germany
• Does Thermal data make the detection systems more reliable?