Cascading Modular Network (CAM-Net) for Multimodal Image Synthesis

Abstract

Deep generative models such as GANs have driven impressive advances in conditional image synthesis in recent years. A persistent challenge has been to generate diverse versions of output images from the same input image, due to the problem of mode collapse: because only one ground truth output image is given per input image, only one mode of the conditional distribution is modelled. In this paper, we focus on this problem of multimodal conditional image synthesis and build on the recently proposed technique of Implicit Maximum Likelihood Estimation (IMLE). Prior IMLE-based methods required different architectures for different tasks, which limit their applicability, and were lacking in fine details in the generated images. We propose CAM-Net, a unified architecture that can be applied to a broad range of tasks. Additionally, it is capable of generating convincing high frequency details, achieving a reduction of the Frechet Inception Distance (FID) by up to 45.3% compared to the baseline.

arXiv commentVideos available as ancillary files

References in corpus (18)

• Auto-Encoding Variational Bayes
• Conditional Generative Adversarial Nets
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Denoising Diffusion Probabilistic Models
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Conditional Image Synthesis With Auxiliary Classifier GANs
• InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
• ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks
• Toward Multimodal Image-to-Image Translation
• Density estimation using Real NVP
• NVAE: A Deep Hierarchical Variational Autoencoder
• Residual Dense Network for Image Super-Resolution
• Learning to Generate Images of Outdoor Scenes from Attributes and Semantic Layouts
• Implicit Generation and Generalization in Energy-Based Models
• Implicit Maximum Likelihood Estimation
• Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images
• Gotta Go Fast When Generating Data with Score-Based Models