On Feature Normalization and Data Augmentation

Abstract

The moments (a.k.a., mean and standard deviation) of latent features are often removed as noise when training image recognition models, to increase stability and reduce training time. However, in the field of image generation, the moments play a much more central role. Studies have shown that the moments extracted from instance normalization and positional normalization can roughly capture style and shape information of an image. Instead of being discarded, these moments are instrumental to the generation process. In this paper we propose Moment Exchange, an implicit data augmentation method that encourages the model to utilize the moment information also for recognition models. Specifically, we replace the moments of the learned features of one training image by those of another, and also interpolate the target labels -- forcing the model to extract training signal from the moments in addition to the normalized features. As our approach is fast, operates entirely in feature space, and mixes different signals than prior methods, one can effectively combine it with existing augmentation approaches. We demonstrate its efficacy across several recognition benchmark data sets where it improves the generalization capability of highly competitive baseline networks with remarkable consistency.

arXiv commentCVPR 2021. Code is available at https://github.com/Boyiliee/MoEx

References in corpus (22)

• Adam: A Method for Stochastic Optimization
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Neural Architecture Search with Reinforcement Learning
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Improved Regularization of Convolutional Neural Networks with Cutout
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• BERTScore: Evaluating Text Generation with BERT
• Wide Residual Networks
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Fast Graph Representation Learning with PyTorch Geometric
• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• DropBlock: A regularization method for convolutional networks
• QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension
• Pay Less Attention with Lightweight and Dynamic Convolutions
• ShakeDrop Regularization for Deep Residual Learning
• Natural Adversarial Examples
• Implicit Semantic Data Augmentation for Deep Networks
• Adversarial Examples Improve Image Recognition
• Generalization in Machine Learning via Analytical Learning Theory
• Shape-Texture Debiased Neural Network Training

Cited by in corpus (15)

• Natural Adversarial Examples
• Review: Deep Learning in Electron Microscopy
• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• A cascade network for Detecting COVID-19 using chest x-rays
• Rethinking "Batch" in BatchNorm
• MatchboxNet: 1D Time-Channel Separable Convolutional Neural Network Architecture for Speech Commands Recognition
• Does enhanced shape bias improve neural network robustness to common corruptions?
• Adversarially Adaptive Normalization for Single Domain Generalization
• Adversarial Feature Augmentation and Normalization for Visual Recognition
• Rethinking Natural Adversarial Examples for Classification Models
• Contrastive ACE: Domain Generalization Through Alignment of Causal Mechanisms
• Uncertainty-Aware Learning for Improvements in Image Quality of the Canada-France-Hawaii Telescope
• Consistency Training of Multi-exit Architectures for Sensor Data
• Regularization via Adaptive Pairwise Label Smoothing
• What Can Style Transfer and Paintings Do For Model Robustness?