Making Convolutional Networks Shift-Invariant Again

Abstract

Modern convolutional networks are not shift-invariant, as small input shifts or translations can cause drastic changes in the output. Commonly used downsampling methods, such as max-pooling, strided-convolution, and average-pooling, ignore the sampling theorem. The well-known signal processing fix is anti-aliasing by low-pass filtering before downsampling. However, simply inserting this module into deep networks degrades performance; as a result, it is seldomly used today. We show that when integrated correctly, it is compatible with existing architectural components, such as max-pooling and strided-convolution. We observe \textit{increased accuracy} in ImageNet classification, across several commonly-used architectures, such as ResNet, DenseNet, and MobileNet, indicating effective regularization. Furthermore, we observe \textit{better generalization}, in terms of stability and robustness to input corruptions. Our results demonstrate that this classical signal processing technique has been undeservingly overlooked in modern deep networks. Code and anti-aliased versions of popular networks are available at https://richzhang.github.io/antialiased-cnns/ .

Accepted to ICML 2019

Cited by in corpus (81)

• Score-Based Generative Modeling through Stochastic Differential Equations
• Alias-Free Generative Adversarial Networks
• AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty
• Leveraging Frequency Analysis for Deep Fake Image Recognition
• StyleNeRF: A Style-based 3D-Aware Generator for High-resolution Image Synthesis
• Why do deep convolutional networks generalize so poorly to small image transformations?
• Measuring Robustness to Natural Distribution Shifts in Image Classification
• Improving robustness against common corruptions by covariate shift adaptation
• Swapping Autoencoder for Deep Image Manipulation
• A New Dataset, Poisson GAN and AquaNet for Underwater Object Grabbing
• Boosting Adversarial Training with Hypersphere Embedding
• Learning Enriched Features for Real Image Restoration and Enhancement
• Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation
• Fast Facial Landmark Detection and Applications: A Survey
• ImageNet-Patch: A Dataset for Benchmarking Machine Learning Robustness against Adversarial Patches
• Deformable Kernels: Adapting Effective Receptive Fields for Object Deformation
• 3D Scene Geometry Estimation from 360${}^{\circ }$ Imagery: A Survey
• Mind the Pad -- CNNs can Develop Blind Spots
• FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizes
• Resolution learning in deep convolutional networks using scale-space theory
• Time-Domain Audio Source Separation Based on Wave-U-Net Combined with Discrete Wavelet Transform
• PCONV: The Missing but Desirable Sparsity in DNN Weight Pruning for Real-time Execution on Mobile Devices
• Delving Deeper into Anti-aliasing in ConvNets
• Investigating Shift-Variance of Convolutional Neural Networks in Ultrasound Image Segmentation
• Knee Injury Detection using MRI with Efficiently-Layered Network (ELNet)
• Does enhanced shape bias improve neural network robustness to common corruptions?
• Self-Diagnosing GAN: Diagnosing Underrepresented Samples in Generative Adversarial Networks
• Robust Image Classification Using A Low-Pass Activation Function and DCT Augmentation
• Volumetric landmark detection with a multi-scale shift equivariant neural network
• Score-based Generative Modeling in Latent Space
• Total Deep Variation: A Stable Regularizer for Inverse Problems
• What Does CNN Shift Invariance Look Like? A Visualization Study
• DexPilot: Vision Based Teleoperation of Dexterous Robotic Hand-Arm System
• PIPAL: a Large-Scale Image Quality Assessment Dataset for Perceptual Image Restoration
• Coordinate Independent Convolutional Networks -- Isometry and Gauge Equivariant Convolutions on Riemannian Manifolds
• Towards Robust Vision Transformer
• A survey on Kornia: an Open Source Differentiable Computer Vision Library for PyTorch
• LiftPool: Bidirectional ConvNet Pooling
• Transfering Low-Frequency Features for Domain Adaptation
• Alias-Free Convnets: Fractional Shift Invariance via Polynomial Activations
• Neural Anisotropy Directions
• Fre-GAN: Adversarial Frequency-consistent Audio Synthesis
• Relative stability toward diffeomorphisms indicates performance in deep nets
• Shift Invariance Can Reduce Adversarial Robustness
• NeRD: Neural Representation of Distribution for Medical Image Segmentation
• PropagationNet: Propagate Points to Curve to Learn Structure Information
• Deep Scattering Network with Max-pooling
• Shift Equivariance in Object Detection
• Unsupervised Change Detection in Satellite Images with Generative Adversarial Network
• Curriculum By Smoothing
• Parallel Residual Bi-Fusion Feature Pyramid Network for Accurate Single-Shot Object Detection
• Improving Sound Event Classification by Increasing Shift Invariance in Convolutional Neural Networks
• RANSAC-Flow: generic two-stage image alignment
• Anti-aliasing Deep Image Classifiers using Novel Depth Adaptive Blurring and Activation Function
• Why Accuracy Is Not Enough: The Need for Consistency in Object Detection
• Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models
• Grounding inductive biases in natural images:invariance stems from variations in data
• Spectral Roll-off Points Variations: Exploring Useful Information in Feature Maps by Its Variations
• Learning Translation Invariance in CNNs
• Data augmentation and image understanding
• Learning to Factorize and Relight a City
• UniNet: Unified Architecture Search with Convolution, Transformer, and MLP
• Revisiting Adaptive Convolutions for Video Frame Interpolation
• What is Learnt by the LEArnable Front-end (LEAF)? Adapting Per-Channel Energy Normalisation (PCEN) to Noisy Conditions
• Frequency Pooling: Shift-Equivalent and Anti-Aliasing Downsampling
• Don't ignore Dropout in Fully Convolutional Networks
• High resolution weakly supervised localization architectures for medical images
• Do CNNs Encode Data Augmentations?
• The art of defense: letting networks fool the attacker
• Modeling the Nonsmoothness of Modern Neural Networks
• Improved Robustness of Vision Transformer via PreLayerNorm in Patch Embedding
• Information Compensation for Deep Conditional Generative Networks
• Autoequivariant Network Search via Group Decomposition
• Augmented Equivariant Attention Networks for Microscopy Image Reconstruction
• Exploring Novel Pooling Strategies for Edge Preserved Feature Maps in Convolutional Neural Networks
• On the Frequency Bias of Generative Models
• Group Equivariant Subsampling
• Neuron segmentation using 3D wavelet integrated encoder-decoder network
• Shape Adaptor: A Learnable Resizing Module
• Learning Equivariant Representations
• How saccadic vision might help with theinterpretability of deep networks