Autoencoding Under Normalization Constraints

Abstract

Likelihood is a standard estimate for outlier detection. The specific role of the normalization constraint is to ensure that the out-of-distribution (OOD) regime has a small likelihood when samples are learned using maximum likelihood. Because autoencoders do not possess such a process of normalization, they often fail to recognize outliers even when they are obviously OOD. We propose the Normalized Autoencoder (NAE), a normalized probabilistic model constructed from an autoencoder. The probability density of NAE is defined using the reconstruction error of an autoencoder, which is differently defined in the conventional energy-based model. In our model, normalization is enforced by suppressing the reconstruction of negative samples, significantly improving the outlier detection performance. Our experimental results confirm the efficacy of NAE, both in detecting outliers and in generating in-distribution samples.

arXiv commentAccepted to ICML 2021. The code is released in https://github.com/swyoon/normalized-autoencoders . The interactive web demo on outlier reconstruction phenomenon and normalized autoencoders can be found in https://swyoon.github.io/outlier-reconstruction

References in corpus (23)

• Adam: A Method for Stochastic Optimization
• Distributed Representations of Words and Phrases and their Compositionality
• Auto-Encoding Variational Bayes
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Spectral Normalization for Generative Adversarial Networks
• Neural Discrete Representation Learning
• Energy-based Generative Adversarial Network
• Improving Unsupervised Defect Segmentation by Applying Structural Similarity to Autoencoders
• Deep Anomaly Detection with Outlier Exposure
• A Theory of Generative ConvNet
• Memorizing Normality to Detect Anomaly: Memory-augmented Deep Autoencoder for Unsupervised Anomaly Detection
• Improved Techniques for Training Score-Based Generative Models
• Generative Probabilistic Novelty Detection with Adversarial Autoencoders
• FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models
• How to Train Your Energy-Based Models
• Likelihood Regret: An Out-of-Distribution Detection Score For Variational Auto-encoder
• Anomaly Detection for Skin Disease Images Using Variational Autoencoder
• Flows for simultaneous manifold learning and density estimation
• Perfect density models cannot guarantee anomaly detection
• Autoregressive Energy Machines
• Latent Variables on Spheres for Autoencoders in High Dimensions
• Fixing Bias in Reconstruction-based Anomaly Detection with Lipschitz Discriminators
• Autoregressive Score Matching