Batch norm with entropic regularization turns deterministic autoencoders into generative models

Abstract

The variational autoencoder is a well defined deep generative model that utilizes an encoder-decoder framework where an encoding neural network outputs a non-deterministic code for reconstructing an input. The encoder achieves this by sampling from a distribution for every input, instead of outputting a deterministic code per input. The great advantage of this process is that it allows the use of the network as a generative model for sampling from the data distribution beyond provided samples for training. We show in this work that utilizing batch normalization as a source for non-determinism suffices to turn deterministic autoencoders into generative models on par with variational ones, so long as we add a suitable entropic regularization to the training objective.

References in corpus (12)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models
• Wasserstein Auto-Encoders
• Lagging Inference Networks and Posterior Collapse in Variational Autoencoders
• Fixup Initialization: Residual Learning Without Normalization
• From Variational to Deterministic Autoencoders
• Norm matters: efficient and accurate normalization schemes in deep networks
• Diagnosing and Enhancing VAE Models
• Implicit Maximum Likelihood Estimation
• Batch Normalization is a Cause of Adversarial Vulnerability
• A Mean Field Theory of Batch Normalization
• Online Normalization for Training Neural Networks