High Frequency Component Helps Explain the Generalization of Convolutional Neural Networks

Abstract

We investigate the relationship between the frequency spectrum of image data and the generalization behavior of convolutional neural networks (CNN). We first notice CNN's ability in capturing the high-frequency components of images. These high-frequency components are almost imperceptible to a human. Thus the observation leads to multiple hypotheses that are related to the generalization behaviors of CNN, including a potential explanation for adversarial examples, a discussion of CNN's trade-off between robustness and accuracy, and some evidence in understanding training heuristics.

arXiv commentTo appear at CVPR 2020 as an Oral paper

References in corpus (29)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• ADADELTA: An Adaptive Learning Rate Method
• mixup: Beyond Empirical Risk Minimization
• Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning
• Theoretically Principled Trade-off between Robustness and Accuracy
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• Adversarial Examples Are Not Bugs, They Are Features
• A Closer Look at Memorization in Deep Networks
• Learning Robust Global Representations by Penalizing Local Predictive Power
• Generating Adversarial Examples with Adversarial Networks
• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data
• Measuring the tendency of CNNs to Learn Surface Statistical Regularities
• Houdini: Fooling Deep Structured Prediction Models
• Sharp Minima Can Generalize For Deep Nets
• Adversarial examples from computational constraints
• Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes
• Sparse DNNs with Improved Adversarial Robustness
• Learning Robust Representations by Projecting Superficial Statistics Out
• On the Origin of Deep Learning
• Deep Defense: Training DNNs with Improved Adversarial Robustness
• Generative Adversarial Trainer: Defense to Adversarial Perturbations with GAN
• Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks
• A Simple Explanation for the Existence of Adversarial Examples with Small Hamming Distance
• Generalizable Adversarial Training via Spectral Normalization
• Are Accuracy and Robustness Correlated?
• Batch Normalization is a Cause of Adversarial Vulnerability
• ADef: an Iterative Algorithm to Construct Adversarial Deformations

Cited by in corpus (6)

• Generalized Out-of-Distribution Detection: A Survey
• Towards Frequency-Based Explanation for Robust CNN
• Word Shape Matters: Robust Machine Translation with Visual Embedding
• Data augmentation and image understanding
• A Singular Value Perspective on Model Robustness
• Identifying Layers Susceptible to Adversarial Attacks