Why do deep convolutional networks generalize so poorly to small image transformations?

Abstract

Convolutional Neural Networks (CNNs) are commonly assumed to be invariant to small image transformations: either because of the convolutional architecture or because they were trained using data augmentation. Recently, several authors have shown that this is not the case: small translations or rescalings of the input image can drastically change the network's prediction. In this paper, we quantify this phenomena and ask why neither the convolutional architecture nor data augmentation are sufficient to achieve the desired invariance. Specifically, we show that the convolutional architecture does not give invariance since architectures ignore the classical sampling theorem, and data augmentation does not give invariance because the CNNs learn to be invariant to transformations only for images that are very similar to typical images from the training set. We discuss two possible solutions to this problem: (1) antialiasing the intermediate representations and (2) increasing data augmentation and show that they provide only a partial solution at best. Taken together, our results indicate that the problem of insuring invariance to small image transformations in neural networks while preserving high accuracy remains unsolved.

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Multi-Scale Context Aggregation by Dilated Convolutions
• Rotation-invariant convolutional neural networks for galaxy morphology prediction
• Synthesizing Robust Adversarial Examples
• Group Equivariant Convolutional Networks
• Robust Physical-World Attacks on Deep Learning Models
• Making Convolutional Networks Shift-Invariant Again
• Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
• Measuring the tendency of CNNs to Learn Surface Statistical Regularities
• Scale-Invariant Convolutional Neural Networks
• Quantifying Translation-Invariance in Convolutional Neural Networks
• Exploring the Space of Black-box Attacks on Deep Neural Networks
• Transformation Properties of Learned Visual Representations
• Polar Transformer Networks
• Geodesics of learned representations
• Query-limited Black-box Attacks to Classifiers
• Studying Invariances of Trained Convolutional Neural Networks

Cited by in corpus (70)

• Shortcut Learning in Deep Neural Networks
• AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
• A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects
• 3D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data
• Leveraging Frequency Analysis for Deep Fake Image Recognition
• Adversarial Examples Are a Natural Consequence of Test Error in Noise
• Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming
• The Origins and Prevalence of Texture Bias in Convolutional Neural Networks
• Predicting the Generalization Gap in Deep Networks with Margin Distributions
• Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations
• Improving Robustness Without Sacrificing Accuracy with Patch Gaussian Augmentation
• SpaceNet MVOI: a Multi-View Overhead Imagery Dataset
• DIVA: Domain Invariant Variational Autoencoders
• A Fourier Perspective on Model Robustness in Computer Vision
• Deep Learning Methods for Solving Linear Inverse Problems: Research Directions and Paradigms
• Adversarial camera stickers: A physical camera-based attack on deep learning systems
• ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction
• Affinity and Diversity: Quantifying Mechanisms of Data Augmentation
• Signal Demodulation with Machine Learning Methods for Physical Layer Visible Light Communications: Prototype Platform, Open Dataset and Algorithms
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
• CNN-generated images are surprisingly easy to spot... for now
• On the "steerability" of generative adversarial networks
• Using Videos to Evaluate Image Model Robustness
• Uncertainty-based Modulation for Lifelong Learning
• Mitigating Bias in Calibration Error Estimation
• AdaMatch: A Unified Approach to Semi-Supervised Learning and Domain Adaptation
• The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning
• Empirical Upper Bound in Object Detection and More
• Investigating Shift-Variance of Convolutional Neural Networks in Ultrasound Image Segmentation
• Characterizing and Avoiding Negative Transfer
• A Little Fog for a Large Turn
• Understanding the Limitations of Conditional Generative Models
• Using learned optimizers to make models robust to input noise
• Wavelet Integrated CNNs for Noise-Robust Image Classification
• Coordinate Independent Convolutional Networks -- Isometry and Gauge Equivariant Convolutions on Riemannian Manifolds
• From Discrete to Continuous Convolution Layers
• Deep Networks from the Principle of Rate Reduction
• Causally motivated Shortcut Removal Using Auxiliary Labels
• Translation Insensitive CNNs
• Towards Resistant Audio Adversarial Examples
• Detecting Overfitting via Adversarial Examples
• Robust neural circuit reconstruction from serial electron microscopy with convolutional recurrent networks
• Relative stability toward diffeomorphisms indicates performance in deep nets
• A Possible Reason for why Data-Driven Beats Theory-Driven Computer Vision
• A Self-Supervised Feature Map Augmentation (FMA) Loss and Combined Augmentations Finetuning to Efficiently Improve the Robustness of CNNs
• Improving Sound Event Classification by Increasing Shift Invariance in Convolutional Neural Networks
• Deep learning for pedestrians: backpropagation in CNNs
• Identifying Model Weakness with Adversarial Examiner
• RAIN: A Simple Approach for Robust and Accurate Image Classification Networks
• Challenges and Prospects in Vision and Language Research
• Noise Optimization for Artificial Neural Networks
• Can 3D Adversarial Logos Cloak Humans?
• On the Ethics of Building AI in a Responsible Manner
• Natural Perturbed Training for General Robustness of Neural Network Classifiers
• Margin-Based Regularization and Selective Sampling in Deep Neural Networks
• Training Artificial Neural Networks by Generalized Likelihood Ratio Method: Exploring Brain-like Learning to Improve Robustness
• Livewired Neural Networks: Making Neurons That Fire Together Wire Together
• Frequency Pooling: Shift-Equivalent and Anti-Aliasing Downsampling
• Learning Transferable 3D Adversarial Cloaks for Deep Trained Detectors
• Computational principles of intelligence: learning and reasoning with neural networks
• Robust Learning with Frequency Domain Regularization
• Improved Robustness of Vision Transformer via PreLayerNorm in Patch Embedding
• Shape Defense Against Adversarial Attacks
• Built-in Elastic Transformations for Improved Robustness
• Exploration into Translation-Equivariant Image Quantization
• Are You Tampering With My Data?
• Wiggling Weights to Improve the Robustness of Classifiers
• Equivariant Point Network for 3D Point Cloud Analysis
• Empirical Upper Bound, Error Diagnosis and Invariance Analysis of Modern Object Detectors
• A Closed-Form Learned Pooling for Deep Classification Networks