Denoising Criterion for Variational Auto-Encoding Framework

Abstract

Denoising autoencoders (DAE) are trained to reconstruct their clean inputs with noise injected at the input level, while variational autoencoders (VAE) are trained with noise injected in their stochastic hidden layer, with a regularizer that encourages this noise injection. In this paper, we show that injecting noise both in input and in the stochastic hidden layer can be advantageous and we propose a modified variational lower bound as an improved objective function in this setup. When input is corrupted, then the standard VAE lower bound involves marginalizing the encoder conditional distribution over the input noise, which makes the training criterion intractable. Instead, we propose a modified training criterion which corresponds to a tractable bound when input is corrupted. Experimentally, we find that the proposed denoising variational autoencoder (DVAE) yields better average log-likelihood than the VAE and the importance weighted autoencoder on the MNIST and Frey Face datasets.

ICLR conference submission

Cited by in corpus (19)

• Recent Advances in Convolutional Neural Networks
• Solving Inverse Problems by Joint Posterior Maximization with Autoencoding Prior
• Educating Text Autoencoders: Latent Representation Guidance via Denoising
• Semi-Implicit Variational Inference
• Learning about learning by many-body systems
• Information Bottleneck and its Applications in Deep Learning
• Blind Image Denoising and Inpainting Using Robust Hadamard Autoencoders
• Spectrum-Guided Adversarial Disparity Learning
• Mask-GVAE: Blind Denoising Graphs via Partition
• Jigsaw-VAE: Towards Balancing Features in Variational Autoencoders
• Do sequence-to-sequence VAEs learn global features of sentences?
• Adversarial and Contrastive Variational Autoencoder for Sequential Recommendation
• Evolutionary Variational Optimization of Generative Models
• Fully Unsupervised Diversity Denoising with Convolutional Variational Autoencoders
• Exploring Versatile Prior for Human Motion via Motion Frequency Guidance
• Learning and Inference in Imaginary Noise Models
• Quantization-Based Regularization for Autoencoders
• Illiterate DALL-E Learns to Compose
• PicTalky: Augmentative and Alternative Communication Software for Language Developmental Disabilities