Hyperspherical Variational Auto-Encoders

Abstract

The Variational Auto-Encoder (VAE) is one of the most used unsupervised machine learning models. But although the default choice of a Gaussian distribution for both the prior and posterior represents a mathematically convenient distribution often leading to competitive results, we show that this parameterization fails to model data with a latent hyperspherical structure. To address this issue we propose using a von Mises-Fisher (vMF) distribution instead, leading to a hyperspherical latent space. Through a series of experiments we show how such a hyperspherical VAE, or $\mathcal{S}$-VAE, is more suitable for capturing data with a hyperspherical latent structure, while outperforming a normal, $\mathcal{N}$-VAE, in low dimensions on other data types. Code at http://github.com/nicola-decao/s-vae-tf and https://github.com/nicola-decao/s-vae-pytorch

arXiv commentCode at http://github.com/nicola-decao/s-vae-tf and https://github.com/nicola-decao/s-vae-pytorch, Blogpost: https://nicola-decao.github.io/s-vae

References in corpus (6)

• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Variational Graph Auto-Encoders
• Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders
• von Mises-Fisher Mixture Model-based Deep learning: Application to Face Verification
• Clustering and Unsupervised Anomaly Detection with L2 Normalized Deep Auto-Encoder Representations
• Riemannian Stein Variational Gradient Descent for Bayesian Inference

Cited by in corpus (53)

• MolGAN: An implicit generative model for small molecular graphs
• Neural Bellman-Ford Networks: A General Graph Neural Network Framework for Link Prediction
• Diagnosing and Enhancing VAE Models
• Semi-Implicit Graph Variational Auto-Encoders
• Continuous Hierarchical Representations with Poincaré Variational Auto-Encoders
• Generative Capacity of Probabilistic Protein Sequence Models
• DDTCDR: Deep Dual Transfer Cross Domain Recommendation
• A Tutorial on Deep Latent Variable Models of Natural Language
• Poincaré Wasserstein Autoencoder
• Structured Variational Inference for Simulating Populations of Radio Galaxies
• Dual Metric Learning for Effective and Efficient Cross-Domain Recommendations
• Preventing Posterior Collapse with delta-VAEs
• Coupling Retrieval and Meta-Learning for Context-Dependent Semantic Parsing
• Latent Variable Modelling with Hyperbolic Normalizing Flows
• Coordinate Independent Convolutional Networks -- Isometry and Gauge Equivariant Convolutions on Riemannian Manifolds
• Variational Autoencoders with Riemannian Brownian Motion Priors
• Radial and Directional Posteriors for Bayesian Neural Networks
• Chart Auto-Encoders for Manifold Structured Data
• Geometry-Aware Hamiltonian Variational Auto-Encoder
• Variational Integrator Networks for Physically Structured Embeddings
• Orientation-Disentangled Unsupervised Representation Learning for Computational Pathology
• tvGP-VAE: Tensor-variate Gaussian Process Prior Variational Autoencoder
• Variational Autoencoder with Implicit Optimal Priors
• Max-Affine Spline Insights into Deep Generative Networks
• Anomaly scores for generative models
• Addressing the Topological Defects of Disentanglement via Distributed Operators
• Increasing Expressivity of a Hyperspherical VAE
• A Batch Normalized Inference Network Keeps the KL Vanishing Away
• Topological Constraints on Homeomorphic Auto-Encoding
• Contrastive Attraction and Contrastive Repulsion for Representation Learning
• Dueling Decoders: Regularizing Variational Autoencoder Latent Spaces
• Neural Variational Inference For Estimating Uncertainty in Knowledge Graph Embeddings
• Disentanglement with Hyperspherical Latent Spaces using Diffusion Variational Autoencoders
• Learning Compact Representations of Neural Networks using DiscriminAtive Masking (DAM)
• Bayes-Factor-VAE: Hierarchical Bayesian Deep Auto-Encoder Models for Factor Disentanglement
• MaGNET: Uniform Sampling from Deep Generative Network Manifolds Without Retraining
• On The Distribution of Penultimate Activations of Classification Networks
• Geodesics in fibered latent spaces: A geometric approach to learning correspondences between conditions
• Copula-like Variational Inference
• Generative Model without Prior Distribution Matching
• A Bayesian Approach to Spherical Factor Analysis for Binary Data
• Momentum Contrastive Autoencoder: Using Contrastive Learning for Latent Space Distribution Matching in WAE
• Learning with Hyperspherical Uniformity
• Memory-Associated Differential Learning
• Maximum likelihood estimation of the Fisher-Bingham distribution via efficient calculation of its normalizing constant
• Hyperbolic Manifold Regression
• Multi network InfoMax: A pre-training method involving graph convolutional networks
• BooVAE: Boosting Approach for Continual Learning of VAE
• Nearest Neighbor Search for Hyperbolic Embeddings
• Not Only Look But Observe: Variational Observation Model of Scene-Level 3D Multi-Object Understanding for Probabilistic SLAM
• Computational Analysis of Deformable Manifolds: from Geometric Modelling to Deep Learning
• Embedding-reparameterization procedure for manifold-valued latent variables in generative models
• A Wasserstein Minimum Velocity Approach to Learning Unnormalized Models