Structural-analogy from a Single Image Pair

Abstract

The task of unsupervised image-to-image translation has seen substantial advancements in recent years through the use of deep neural networks. Typically, the proposed solutions learn the characterizing distribution of two large, unpaired collections of images, and are able to alter the appearance of a given image, while keeping its geometry intact. In this paper, we explore the capabilities of neural networks to understand image structure given only a single pair of images, A and B. We seek to generate images that are structurally aligned: that is, to generate an image that keeps the appearance and style of B, but has a structural arrangement that corresponds to A. The key idea is to map between image patches at different scales. This enables controlling the granularity at which analogies are produced, which determines the conceptual distinction between style and content. In addition to structural alignment, our method can be used to generate high quality imagery in other conditional generation tasks utilizing images A and B only: guided image synthesis, style and texture transfer, text translation as well as video translation. Our code and additional results are available in https://github.com/rmokady/structural-analogy/.

arXiv commentPublished in 'Computer Graphics Forum'

References in corpus (14)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Toward Multimodal Image-to-Image Translation
• Deep Image Prior
• Learning to Discover Cross-Domain Relations with Generative Adversarial Networks
• Universal Style Transfer via Feature Transforms
• Word Translation Without Parallel Data
• Unsupervised Learning of Disentangled Representations from Video
• Non-Stationary Texture Synthesis by Adversarial Expansion
• DRIT++: Diverse Image-to-Image Translation via Disentangled Representations
• Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observations
• Demystifying Inter-Class Disentanglement
• TuiGAN: Learning Versatile Image-to-Image Translation with Two Unpaired Images

Cited by in corpus (6)

• Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications
• SeamlessGAN: Self-Supervised Synthesis of Tileable Texture Maps
• Disentangling Structure and Appearance in ViT Feature Space
• TuiGAN: Learning Versatile Image-to-Image Translation with Two Unpaired Images
• Neural Photometry-guided Visual Attribute Transfer
• TransferI2I: Transfer Learning for Image-to-Image Translation from Small Datasets