Deep Automodulators

Abstract

We introduce a new category of generative autoencoders called automodulators. These networks can faithfully reproduce individual real-world input images like regular autoencoders, but also generate a fused sample from an arbitrary combination of several such images, allowing instantaneous 'style-mixing' and other new applications. An automodulator decouples the data flow of decoder operations from statistical properties thereof and uses the latent vector to modulate the former by the latter, with a principled approach for mutual disentanglement of decoder layers. Prior work has explored similar decoder architecture with GANs, but their focus has been on random sampling. A corresponding autoencoder could operate on real input images. For the first time, we show how to train such a general-purpose model with sharp outputs in high resolution, using novel training techniques, demonstrated on four image data sets. Besides style-mixing, we show state-of-the-art results in autoencoder comparison, and visual image quality nearly indistinguishable from state-of-the-art GANs. We expect the automodulator variants to become a useful building block for image applications and other data domains.

arXiv commentTo appear in Advances in Neural Information Processing Systems (NeurIPS 2020)

References in corpus (14)

• Auto-Encoding Variational Bayes
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Instance Normalization: The Missing Ingredient for Fast Stylization
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
• A Learned Representation For Artistic Style
• BEGAN: Boundary Equilibrium Generative Adversarial Networks
• Large Scale Adversarial Representation Learning
• IntroVAE: Introspective Variational Autoencoders for Photographic Image Synthesis
• Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks
• One-Shot Generalization in Deep Generative Models
• On Self Modulation for Generative Adversarial Networks