Texture Synthesis with Spatial Generative Adversarial Networks

Abstract

Generative adversarial networks (GANs) are a recent approach to train generative models of data, which have been shown to work particularly well on image data. In the current paper we introduce a new model for texture synthesis based on GAN learning. By extending the input noise distribution space from a single vector to a whole spatial tensor, we create an architecture with properties well suited to the task of texture synthesis, which we call spatial GAN (SGAN). To our knowledge, this is the first successful completely data-driven texture synthesis method based on GANs. Our method has the following features which make it a state of the art algorithm for texture synthesis: high image quality of the generated textures, very high scalability w.r.t. the output texture size, fast real-time forward generation, the ability to fuse multiple diverse source images in complex textures. To illustrate these capabilities we present multiple experiments with different classes of texture images and use cases. We also discuss some limitations of our method with respect to the types of texture images it can synthesize, and compare it to other neural techniques for texture generation.

presented at the NIPS 2016 adversarial learning workshop, Barcelona, Spain

References in corpus (1)

• Texture Networks: Feed-forward Synthesis of Textures and Stylized Images

Cited by in corpus (44)

• Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy
• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• Image De-raining Using a Conditional Generative Adversarial Network
• Non-Stationary Texture Synthesis by Adversarial Expansion
• Generative Adversarial Networks (GANs) in Networking: A Comprehensive Survey & Evaluation
• Deep generative models in inversion: a review and development of a new approach based on a variational autoencoder
• Local Aggregation for Unsupervised Learning of Visual Embeddings
• Gradient-based deterministic inversion of geophysical data with Generative Adversarial Networks: is it feasible?
• Recent Developments Combining Ensemble Smoother and Deep Generative Networks for Facies History Matching
• TileGAN: Synthesis of Large-Scale Non-Homogeneous Textures
• Attributing Fake Images to GANs: Learning and Analyzing GAN Fingerprints
• InGAN: Capturing and Remapping the "DNA" of a Natural Image
• Language Guided Fashion Image Manipulation with Feature-wise Transformations
• DeepFlow: History Matching in the Space of Deep Generative Models
• TextureGAN: Controlling Deep Image Synthesis with Texture Patches
• Generative Adversarial Networks: A Survey Towards Private and Secure Applications
• Learning Texture Manifolds with the Periodic Spatial GAN
• Fast and Scalable Earth Texture Synthesis using Spatially Assembled Generative Adversarial Neural Networks
• Conditioning of three-dimensional generative adversarial networks for pore and reservoir-scale models
• GramGAN: Deep 3D Texture Synthesis From 2D Exemplars
• RG-Flow: A hierarchical and explainable flow model based on renormalization group and sparse prior
• Adaptive sequential Monte Carlo for posterior inference and model selection among complex geological priors
• Using deep generative neural networks to account for model errors in Markov chain Monte Carlo inversion
• Texture Mixer: A Network for Controllable Synthesis and Interpolation of Texture
• GANosaic: Mosaic Creation with Generative Texture Manifolds
• Transposer: Universal Texture Synthesis Using Feature Maps as Transposed Convolution Filter
• TuiGAN: Learning Versatile Image-to-Image Translation with Two Unpaired Images
• Learning to Synthesize Fashion Textures
• First Order Generative Adversarial Networks
• Learning Neurosymbolic Generative Models via Program Synthesis
• Copy the Old or Paint Anew? An Adversarial Framework for (non-) Parametric Image Stylization
• Transform the Set: Memory Attentive Generation of Guided and Unguided Image Collages
• GMM-Based Generative Adversarial Encoder Learning
• Auto-Embedding Generative Adversarial Networks for High Resolution Image Synthesis
• Intelligent Home 3D: Automatic 3D-House Design from Linguistic Descriptions Only
• Dilated Spatial Generative Adversarial Networks for Ergodic Image Generation
• SETGAN: Scale and Energy Trade-off GANs for Image Applications on Mobile Platforms
• Interpreting Spatially Infinite Generative Models
• Texture Mixing by Interpolating Deep Statistics via Gaussian Models
• Synthesis of High-Quality Visible Faces from Polarimetric Thermal Faces using Generative Adversarial Networks
• Macrocanonical Models for Texture Synthesis
• Texture synthesis via projection onto multiscale, multilayer statistics
• Generative Adversarial Networks for Astronomical Images Generation
• Learning to Transfer Visual Effects from Videos to Images