Associative Compression Networks for Representation Learning

Abstract

This paper introduces Associative Compression Networks (ACNs), a new framework for variational autoencoding with neural networks. The system differs from existing variational autoencoders (VAEs) in that the prior distribution used to model each code is conditioned on a similar code from the dataset. In compression terms this equates to sequentially transmitting the dataset using an ordering determined by proximity in latent space. Since the prior need only account for local, rather than global variations in the latent space, the coding cost is greatly reduced, leading to rich, informative codes. Crucially, the codes remain informative when powerful, autoregressive decoders are used, which we argue is fundamentally difficult with normal VAEs. Experimental results on MNIST, CIFAR-10, ImageNet and CelebA show that ACNs discover high-level latent features such as object class, writing style, pose and facial expression, which can be used to cluster and classify the data, as well as to generate diverse and convincing samples. We conclude that ACNs are a promising new direction for representation learning: one that steps away from IID modelling, and towards learning a structured description of the dataset as a whole.

arXiv commentRevised to clarify difference between ACN and IID loss

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Neural Discrete Representation Learning
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Pixel Recurrent Neural Networks
• InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
• DRAW: A Recurrent Neural Network For Image Generation
• Matching Networks for One Shot Learning
• Adversarial Feature Learning
• Unsupervised Visual Representation Learning by Context Prediction
• Embedding Projector: Interactive Visualization and Interpretation of Embeddings
• Variational inference for Monte Carlo objectives
• Towards Conceptual Compression
• PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications
• VAE with a VampPrior
• PixelSNAIL: An Improved Autoregressive Generative Model
• Discrete Variational Autoencoders
• Online Learning with Gated Linear Networks

Cited by in corpus (10)

• Lifelong Generative Modeling
• Deep Encoder-Decoder Models for Unsupervised Learning of Controllable Speech Synthesis
• The Functional Neural Process
• Preventing Posterior Collapse with delta-VAEs
• What Is Considered Complete for Visual Recognition?
• SCHA-VAE: Hierarchical Context Aggregation for Few-Shot Generation
• Neural Communication Systems with Bandwidth-limited Channel
• No Representation without Transformation
• A Factorial Mixture Prior for Compositional Deep Generative Models
• Kanerva++: extending The Kanerva Machine with differentiable, locally block allocated latent memory