On denoising autoencoders trained to minimise binary cross-entropy

Abstract

Denoising autoencoders (DAEs) are powerful deep learning models used for feature extraction, data generation and network pre-training. DAEs consist of an encoder and decoder which may be trained simultaneously to minimise a loss (function) between an input and the reconstruction of a corrupted version of the input. There are two common loss functions used for training autoencoders, these include the mean-squared error (MSE) and the binary cross-entropy (BCE). When training autoencoders on image data a natural choice of loss function is BCE, since pixel values may be normalised to take values in [0,1] and the decoder model may be designed to generate samples that take values in (0,1). We show theoretically that DAEs trained to minimise BCE may be used to take gradient steps in the data space towards regions of high probability under the data-generating distribution. Previously this had only been shown for DAEs trained using MSE. As a consequence of the theory, iterative application of a trained DAE moves a data sample from regions of low probability to regions of higher probability under the data-generating distribution. Firstly, we validate the theory by showing that novel data samples, consistent with the training data, may be synthesised when the initial data samples are random noise. Secondly, we motivate the theory by showing that initial data samples synthesised via other methods may be improved via iterative application of a trained DAE to those initial samples.

arXiv commentSubmitted to Pattern Recognition Letters

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Auto-Encoding Variational Bayes
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Semi-Supervised Learning with Ladder Networks
• Markov Chain Monte Carlo and Variational Inference: Bridging the Gap
• Towards Principled Methods for Training Generative Adversarial Networks
• Variational Inference using Implicit Distributions
• Variational Generative Stochastic Networks with Collaborative Shaping

Cited by in corpus (6)

• SIXray : A Large-scale Security Inspection X-ray Benchmark for Prohibited Item Discovery in Overlapping Images
• A Bayesian approach to tissue-fraction estimation for oncological PET segmentation
• A tissue-fraction estimation-based segmentation method for quantitative dopamine transporter SPECT
• LioNets: A Neural-Specific Local Interpretation Technique Exploiting Penultimate Layer Information
• AMAD: Adversarial Multiscale Anomaly Detection on High-Dimensional and Time-Evolving Categorical Data
• Automated Human Claustrum Segmentation using Deep Learning Technologies