Regularizing activations in neural networks via distribution matching with the Wasserstein metric

Abstract

Regularization and normalization have become indispensable components in training deep neural networks, resulting in faster training and improved generalization performance. We propose the projected error function regularization loss (PER) that encourages activations to follow the standard normal distribution. PER randomly projects activations onto one-dimensional space and computes the regularization loss in the projected space. PER is similar to the Pseudo-Huber loss in the projected space, thus taking advantage of both $L^1$ and $L^2$ regularization losses. Besides, PER can capture the interaction between hidden units by projection vector drawn from a unit sphere. By doing so, PER minimizes the upper bound of the Wasserstein distance of order one between an empirical distribution of activations and the standard normal distribution. To the best of the authors' knowledge, this is the first work to regularize activations via distribution matching in the probability distribution space. We evaluate the proposed method on the image classification task and the word-level language modeling task.

arXiv commentICLR 2020

References in corpus (16)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Convolutional Sequence to Sequence Learning
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• DropBlock: A regularization method for convolutional networks
• Learning with a Wasserstein Loss
• Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
• Universal representations:The missing link between faces, text, planktons, and cat breeds
• Norm matters: efficient and accurate normalization schemes in deep networks
• The Shattered Gradients Problem: If resnets are the answer, then what is the question?
• Natural Neural Networks
• Decorrelated Batch Normalization
• Revisiting Activation Regularization for Language RNNs
• Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization
• Policy Optimization as Wasserstein Gradient Flows
• Streaming Normalization: Towards Simpler and More Biologically-plausible Normalizations for Online and Recurrent Learning
• Mode Normalization

Cited by in corpus (3)

• Normalization Techniques in Training DNNs: Methodology, Analysis and Application
• Revisiting Explicit Regularization in Neural Networks for Well-Calibrated Predictive Uncertainty
• Topologically Densified Distributions