Evolving Normalization-Activation Layers

Abstract

Normalization layers and activation functions are fundamental components in deep networks and typically co-locate with each other. Here we propose to design them using an automated approach. Instead of designing them separately, we unify them into a single tensor-to-tensor computation graph, and evolve its structure starting from basic mathematical functions. Examples of such mathematical functions are addition, multiplication and statistical moments. The use of low-level mathematical functions, in contrast to the use of high-level modules in mainstream NAS, leads to a highly sparse and large search space which can be challenging for search methods. To address the challenge, we develop efficient rejection protocols to quickly filter out candidate layers that do not work well. We also use multi-objective evolution to optimize each layer's performance across many architectures to prevent overfitting. Our method leads to the discovery of EvoNorms, a set of new normalization-activation layers with novel, and sometimes surprising structures that go beyond existing design patterns. For example, some EvoNorms do not assume that normalization and activation functions must be applied sequentially, nor need to center the feature maps, nor require explicit activation functions. Our experiments show that EvoNorms work well on image classification models including ResNets, MobileNets and EfficientNets but also transfer well to Mask R-CNN with FPN/SpineNet for instance segmentation and to BigGAN for image synthesis, outperforming BatchNorm and GroupNorm based layers in many cases.

References in corpus (20)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Neural Architecture Search with Reinforcement Learning
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Gaussian Error Linear Units (GELUs)
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Neural Architecture Search: A Survey
• Efficient Neural Architecture Search via Parameter Sharing
• Neural Architecture Optimization
• Designing Neural Network Architectures using Reinforcement Learning
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• SNAS: Stochastic Neural Architecture Search
• Random Search and Reproducibility for Neural Architecture Search
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• Differentiable Learning-to-Normalize via Switchable Normalization
• ReZero is All You Need: Fast Convergence at Large Depth
• AutoML-Zero: Evolving Machine Learning Algorithms From Scratch
• Norm matters: efficient and accurate normalization schemes in deep networks
• Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks

Cited by in corpus (10)

• Discovering Parametric Activation Functions
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• Loss Function Discovery for Object Detection via Convergence-Simulation Driven Search
• Smooth activations and reproducibility in deep networks
• Making EfficientNet More Efficient: Exploring Batch-Independent Normalization, Group Convolutions and Reduced Resolution Training
• The Nonlinearity Coefficient - A Practical Guide to Neural Architecture Design
• Learning to Act through Evolution of Neural Diversity in Random Neural Networks
• Improving Task-Agnostic BERT Distillation with Layer Mapping Search
• Deep Preset: Blending and Retouching Photos with Color Style Transfer
• Kanerva++: extending The Kanerva Machine with differentiable, locally block allocated latent memory