Generalized Denoising Auto-Encoders as Generative Models
arXiv:1305.6663
Abstract
Recent work has shown how denoising and contractive autoencoders implicitly capture the structure of the data-generating density, in the case where the corruption noise is Gaussian, the reconstruction error is the squared error, and the data is continuous-valued. This has led to various proposals for sampling from this implicitly learned density function, using Langevin and Metropolis-Hastings MCMC. However, it remained unclear how to connect the training procedure of regularized auto-encoders to the implicit estimation of the underlying data-generating distribution when the data are discrete, or using other forms of corruption process and reconstruction errors. Another issue is the mathematical justification which is only valid in the limit of small corruption noise. We propose here a different attack on the problem, which deals with all these issues: arbitrary (but noisy enough) corruption, arbitrary reconstruction loss (seen as a log-likelihood), handling both discrete and continuous-valued variables, and removing the bias due to non-infinitesimal corruption noise (or non-infinitesimal contractive penalty).
References in corpus (4)
Cited by in corpus (47)
- Deep learning for time series classification: a review
- Tutorial on Variational Autoencoders
- Generative Modeling by Estimating Gradients of the Data Distribution
- Semi-Supervised Learning with Ladder Networks
- Generative Moment Matching Networks
- Deep Generative Stochastic Networks Trainable by Backprop
- Towards Biologically Plausible Deep Learning
- Personalized Machine Learning for Robot Perception of Affect and Engagement in Autism Therapy
- Test-Time Adaptable Neural Networks for Robust Medical Image Segmentation
- Learning Contextual Dependencies with Convolutional Hierarchical Recurrent Neural Networks
- Noise Learning Based Denoising Autoencoder
- Learning Generative Models with Visual Attention
- Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
- Deconstructing the Ladder Network Architecture
- Multi-view Generative Adversarial Networks
- Stabilizing Invertible Neural Networks Using Mixture Models
- Educating Text Autoencoders: Latent Representation Guidance via Denoising
- Learning Neural Generative Dynamics for Molecular Conformation Generation
- Learning to solve the credit assignment problem
- Bidirectional Recurrent Neural Networks as Generative Models - Reconstructing Gaps in Time Series
- Human Motion Prediction via Spatio-Temporal Inpainting
- A Survey on Bayesian Deep Learning
- Learning Energy-Based Models by Diffusion Recovery Likelihood
- Solving Linear Inverse Problems Using the Prior Implicit in a Denoiser
- Perceptual Generative Autoencoders
- Learning the Redundancy-free Features for Generalized Zero-Shot Object Recognition
- Score-based Generative Modeling in Latent Space
- Information Bottleneck and its Applications in Deep Learning
- Variational Prototyping-Encoder: One-Shot Learning with Prototypical Images
- Predicting distributions with Linearizing Belief Networks
- Generative Models For Deep Learning with Very Scarce Data
- Temporal Autoencoding Improves Generative Models of Time Series
- The Labeling Distribution Matrix (LDM): A Tool for Estimating Machine Learning Algorithm Capacity
- Learning Dynamics of Linear Denoising Autoencoders
- Spurious samples in deep generative models: bug or feature?
- An Unsupervised Bayesian Neural Network for Truth Discovery in Social Networks
- GSNs : Generative Stochastic Networks
- Normal-bundle Bootstrap
- Cascading Denoising Auto-Encoder as a Deep Directed Generative Model
- Masking schemes for universal marginalisers
- Momentum Contrastive Autoencoder: Using Contrastive Learning for Latent Space Distribution Matching in WAE
- On the Generative Utility of Cyclic Conditionals
- Piece-wise Matching Layer in Representation Learning for ECG Classification
- Langevin Cooling for Domain Translation
- From Persistent Homology to Reinforcement Learning with Applications for Retail Banking
- What are Neural Networks made of?
- Visualization of AE's Training on Credit Card Transactions with Persistent Homology