General Probabilistic Surface Optimization and Log Density Estimation

Abstract

In this paper we contribute a novel algorithm family, which generalizes many unsupervised techniques including unnormalized and energy models, and allows us to infer different statistical modalities (e.g. data likelihood and ratio between densities) from data samples. The proposed unsupervised technique, named Probabilistic Surface Optimization (PSO), views a model as a flexible surface which can be pushed according to loss-specific virtual stochastic forces, where a dynamical equilibrium is achieved when the pointwise forces on the surface become equal. Concretely, the surface is pushed up and down at points sampled from two different distributions. The averaged up and down forces become functions of these two distribution densities and of force magnitudes defined by the loss of a particular PSO instance. Upon convergence, the force equilibrium imposes an optimized model to be equal to various statistical functions depending on the used magnitude functions. Furthermore, this dynamical-statistical equilibrium is extremely intuitive and useful, providing many implications and possible usages in probabilistic inference. We connect PSO to numerous existing statistical works which are also PSO instances, and derive new PSO-based inference methods as demonstration of PSO exceptional usability. Likewise, based on the insights coming from the virtual-force perspective we analyze PSO stability and propose new ways to improve it. Finally, we present new instances of PSO, termed PSO-LDE, for data log-density estimation and also provide a new NN block-diagonal architecture for increased surface flexibility, which significantly improves estimation accuracy. Both PSO-LDE and the new architecture are combined together as a new density estimation technique. In our experiments we demonstrate this technique to be superior over state-of-the-art baselines in density estimation task for multimodal 20D data.

References in corpus (25)

• Conditional Generative Adversarial Nets
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Spectral Normalization for Generative Adversarial Networks
• The Effectiveness of Data Augmentation in Image Classification using Deep Learning
• NICE: Non-linear Independent Components Estimation
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Variational Inference with Normalizing Flows
• Energy-based Generative Adversarial Network
• Density estimation using Real NVP
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• Game theory, maximum entropy, minimum discrepancy and robust Bayesian decision theory
• MINE: Mutual Information Neural Estimation
• MADE: Masked Autoencoder for Distribution Estimation
• Implicit Regularization in Nonconvex Statistical Estimation: Gradient Descent Converges Linearly for Phase Retrieval, Matrix Completion, and Blind Deconvolution
• Learning in Implicit Generative Models
• Composite Binary Losses
• On the regularization of Wasserstein GANs
• The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies
• RNADE: The real-valued neural autoregressive density-estimator
• High-Dimensional Probability Estimation with Deep Density Models
• Deep Energy Estimator Networks
• Learning Deep Energy Models: Contrastive Divergence vs. Amortized MLE
• Understanding the Effectiveness of Lipschitz-Continuity in Generative Adversarial Nets
• Analysis of Noise Contrastive Estimation from the Perspective of Asymptotic Variance
• Deep PDF: Probabilistic Surface Optimization and Density Estimation