Using Videos to Evaluate Image Model Robustness

Abstract

Human visual systems are robust to a wide range of image transformations that are challenging for artificial networks. We present the first study of image model robustness to the minute transformations found across video frames, which we term "natural robustness". Compared to previous studies on adversarial examples and synthetic distortions, natural robustness captures a more diverse set of common image transformations that occur in the natural environment. Our study across a dozen model architectures shows that more accurate models are more robust to natural transformations, and that robustness to synthetic color distortions is a good proxy for natural robustness. In examining brittleness in videos, we find that majority of the brittleness found in videos lies outside the typical definition of adversarial examples (99.9\%). Finally, we investigate training techniques to reduce brittleness and find that no single technique systematically improves natural robustness across twelve tested architectures.

Video Robustness Dataset included in directory

References in corpus (8)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Intriguing properties of neural networks
• AutoAugment: Learning Augmentation Policies from Data
• Generalisation in humans and deep neural networks
• Why do deep convolutional networks generalize so poorly to small image transformations?
• Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations
• Traffic Signs in the Wild: Highlights from the IEEE Video and Image Processing Cup 2017 Student Competition [SP Competitions]

Cited by in corpus (12)

• Self-training with Noisy Student improves ImageNet classification
• Measuring Robustness to Natural Distribution Shifts in Image Classification
• Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentation
• AdaMatch: A Unified Approach to Semi-Supervised Learning and Domain Adaptation
• The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning
• An Analysis of Deep Object Detectors For Diver Detection
• Recent Advances in Understanding Adversarial Robustness of Deep Neural Networks
• Shift Equivariance in Object Detection
• Identifying Model Weakness with Adversarial Examiner
• Why Accuracy Is Not Enough: The Need for Consistency in Object Detection
• Improving the robustness of ImageNet classifiers using elements of human visual cognition
• Contemplating real-world object classification