Do Normalization Layers in a Deep ConvNet Really Need to Be Distinct?
arXiv:1811.07727
Abstract
Yes, they do. This work investigates a perspective for deep learning: whether different normalization layers in a ConvNet require different normalizers. This is the first step towards understanding this phenomenon. We allow each convolutional layer to be stacked before a switchable normalization (SN) that learns to choose a normalizer from a pool of normalization methods. Through systematic experiments in ImageNet, COCO, Cityscapes, and ADE20K, we answer three questions: (a) Is it useful to allow each normalization layer to select its own normalizer? (b) What impacts the choices of normalizers? (c) Do different tasks and datasets prefer different normalizers? Our results suggest that (1) using distinct normalizers improves both learning and generalization of a ConvNet; (2) the choices of normalizers are more related to depth and batch size, but less relevant to parameter initialization, learning rate decay, and solver; (3) different tasks and datasets have different behaviors when learning to select normalizers.
arXiv commentPreprint. Work in Progress. 14 pages, 13 figures
References in corpus (20)
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- Deep Residual Learning for Image Recognition
- Instance Normalization: The Missing Ingredient for Fast Stylization
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- SGDR: Stochastic Gradient Descent with Warm Restarts
- Going Deeper with Convolutions
- DARTS: Differentiable Architecture Search
- Understanding deep learning requires rethinking generalization
- The Cityscapes Dataset for Semantic Urban Scene Understanding
- Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
- Efficient Neural Architecture Search via Parameter Sharing
- Rethinking the Inception Architecture for Computer Vision
- Layer Normalization
- Gradient-based Hyperparameter Optimization through Reversible Learning
- Pyramid Scene Parsing Network
- Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
- Semantic Understanding of Scenes through the ADE20K Dataset
- Differentiable Learning-to-Normalize via Switchable Normalization
- Do Deep Convolutional Nets Really Need to be Deep and Convolutional?
- Normalizing the Normalizers: Comparing and Extending Network Normalization Schemes