Differentiable Learning-to-Normalize via Switchable Normalization

Abstract

We address a learning-to-normalize problem by proposing Switchable Normalization (SN), which learns to select different normalizers for different normalization layers of a deep neural network. SN employs three distinct scopes to compute statistics (means and variances) including a channel, a layer, and a minibatch. SN switches between them by learning their importance weights in an end-to-end manner. It has several good properties. First, it adapts to various network architectures and tasks (see Fig.1). Second, it is robust to a wide range of batch sizes, maintaining high performance even when small minibatch is presented (e.g. 2 images/GPU). Third, SN does not have sensitive hyper-parameter, unlike group normalization that searches the number of groups as a hyper-parameter. Without bells and whistles, SN outperforms its counterparts on various challenging benchmarks, such as ImageNet, COCO, CityScapes, ADE20K, and Kinetics. Analyses of SN are also presented. We hope SN will help ease the usage and understand the normalization techniques in deep learning. The code of SN has been made available in https://github.com/switchablenorms/.

arXiv commentInternational Conference on Learning Representations (ICLR)

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Instance Normalization: The Missing Ingredient for Fast Stylization
• The Kinetics Human Action Video Dataset
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Efficient Neural Architecture Search via Parameter Sharing
• Feature Pyramid Networks for Object Detection
• Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
• FiLM: Visual Reasoning with a General Conditioning Layer
• Towards Understanding Regularization in Batch Normalization
• Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes
• MegDet: A Large Mini-Batch Object Detector
• Batch Kalman Normalization: Towards Training Deep Neural Networks with Micro-Batches
• SSN: Learning Sparse Switchable Normalization via SparsestMax
• Do Normalization Layers in a Deep ConvNet Really Need to Be Distinct?

Cited by in corpus (35)

• Micro-Batch Training with Batch-Channel Normalization and Weight Standardization
• Review: Deep Learning in Electron Microscopy
• Automatic Ischemic Stroke Lesion Segmentation from Computed Tomography Perfusion Images by Image Synthesis and Attention-Based Deep Neural Networks
• RDP-Net: Region Detail Preserving Network for Change Detection
• Evolving Normalization-Activation Layers
• RelGAN: Multi-Domain Image-to-Image Translation via Relative Attributes
• Four Things Everyone Should Know to Improve Batch Normalization
• Learning to Optimize Domain Specific Normalization for Domain Generalization
• Thickened 2D Networks for Efficient 3D Medical Image Segmentation
• AIM 2020 Challenge on Learned Image Signal Processing Pipeline
• Cross-Iteration Batch Normalization
• Adapted Center and Scale Prediction: More Stable and More Accurate
• Generative Model-Based Ischemic Stroke Lesion Segmentation
• Iterative Normalization: Beyond Standardization towards Efficient Whitening
• Extended Batch Normalization
• Unsupervised Multi-Source Domain Adaptation for Person Re-Identification
• SSN: Learning Sparse Switchable Normalization via SparsestMax
• Positional Normalization
• Rethinking Normalization and Elimination Singularity in Neural Networks
• Deep Residual 3D U-Net for Joint Segmentation and Texture Classification of Nodules in Lung
• Do Normalization Layers in a Deep ConvNet Really Need to Be Distinct?
• The 1st Agriculture-Vision Challenge: Methods and Results
• Channel Equilibrium Networks for Learning Deep Representation
• Towards Understanding Normalization in Neural ODEs
• Instance Enhancement Batch Normalization: an Adaptive Regulator of Batch Noise
• Object Detection from Scratch with Deep Supervision
• Understanding the Effects of Pre-Training for Object Detectors via Eigenspectrum
• Scale Calibrated Training: Improving Generalization of Deep Networks via Scale-Specific Normalization
• Understanding the Disharmony between Weight Normalization Family and Weight Decay: $\epsilon -$shifted $L_2$ Regularizer
• Reducing the feature divergence of RGB and near-infrared images using Switchable Normalization
• An Investigation into the Stochasticity of Batch Whitening
• Group Whitening: Balancing Learning Efficiency and Representational Capacity
• Learning Connectivity of Neural Networks from a Topological Perspective
• LocalNorm: Robust Image Classification through Dynamically Regularized Normalization
• A novel generative reverse net assisted evolution algorithm for expensive-computational optimizations