Gradient Origin Networks
arXiv:2007.02798
Abstract
This paper proposes a new type of generative model that is able to quickly learn a latent representation without an encoder. This is achieved using empirical Bayes to calculate the expectation of the posterior, which is implemented by initialising a latent vector with zeros, then using the gradient of the log-likelihood of the data with respect to this zero vector as new latent points. The approach has similar characteristics to autoencoders, but with a simpler architecture, and is demonstrated in a variational autoencoder equivalent that permits sampling. This also allows implicit representation networks to learn a space of implicit functions without requiring a hypernetwork, retaining their representation advantages across datasets. The experiments show that the proposed method converges faster, with significantly lower reconstruction error than autoencoders, while requiring half the parameters.
arXiv comment16 pages, 17 figures, accepted at ICLR 2021, camera-ready version
References in corpus (15)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Auto-Encoding Variational Bayes
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Learning to Plan Chemical Syntheses
- Pixel Recurrent Neural Networks
- LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
- Variational Inference with Normalizing Flows
- Generative Modeling by Estimating Gradients of the Data Distribution
- Variational Lossy Autoencoder
- Towards Deeper Understanding of Variational Autoencoding Models
- Implicit Generation and Generalization in Energy-Based Models
- MetaSDF: Meta-learning Signed Distance Functions
- Alternating Back-Propagation for Generator Network
- On approximating with neural networks