Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks

Abstract

This paper proposes Markovian Generative Adversarial Networks (MGANs), a method for training generative neural networks for efficient texture synthesis. While deep neural network approaches have recently demonstrated remarkable results in terms of synthesis quality, they still come at considerable computational costs (minutes of run-time for low-res images). Our paper addresses this efficiency issue. Instead of a numerical deconvolution in previous work, we precompute a feed-forward, strided convolutional network that captures the feature statistics of Markovian patches and is able to directly generate outputs of arbitrary dimensions. Such network can directly decode brown noise to realistic texture, or photos to artistic paintings. With adversarial training, we obtain quality comparable to recent neural texture synthesis methods. As no optimization is required any longer at generation time, our run-time performance (0.25M pixel images at 25Hz) surpasses previous neural texture synthesizers by a significant margin (at least 500 times faster). We apply this idea to texture synthesis, style transfer, and video stylization.

arXiv comment17 pages, 15 figures

References in corpus (14)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Striving for Simplicity: The All Convolutional Net
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• Pixel Recurrent Neural Networks
• Understanding Neural Networks Through Deep Visualization
• DRAW: A Recurrent Neural Network For Image Generation
• Autoencoding beyond pixels using a learned similarity metric
• A Theory of Generative ConvNet
• Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks
• Deep Learning Face Attributes in the Wild
• Attribute2Image: Conditional Image Generation from Visual Attributes
• Understanding Deep Image Representations by Inverting Them
• Generating Images from Captions with Attention

Cited by in corpus (7)

• Mode Regularized Generative Adversarial Networks
• Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis
• StyleBank: An Explicit Representation for Neural Image Style Transfer
• Coherent Online Video Style Transfer
• Visual Attribute Transfer through Deep Image Analogy
• Reconstructing Images of Two Adjacent Objects through Scattering Medium Using Generative Adversarial Network
• Selective Sampling and Mixture Models in Generative Adversarial Networks