Combining Markov Random Fields and Convolutional Neural Networks for Image Synthesis

Abstract

This paper studies a combination of generative Markov random field (MRF) models and discriminatively trained deep convolutional neural networks (dCNNs) for synthesizing 2D images. The generative MRF acts on higher-levels of a dCNN feature pyramid, controling the image layout at an abstract level. We apply the method to both photographic and non-photo-realistic (artwork) synthesis tasks. The MRF regularizer prevents over-excitation artifacts and reduces implausible feature mixtures common to previous dCNN inversion approaches, permitting synthezing photographic content with increased visual plausibility. Unlike standard MRF-based texture synthesis, the combined system can both match and adapt local features with considerable variability, yielding results far out of reach of classic generative MRF methods.

9 pages, 9 figures

References in corpus (1)

• Very Deep Convolutional Networks for Large-Scale Image Recognition

Cited by in corpus (18)

• Generating Handwritten Chinese Characters using CycleGAN
• Deep Photo Style Transfer
• Perceptual Generative Adversarial Networks for Small Object Detection
• Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis
• StyleBank: An Explicit Representation for Neural Image Style Transfer
• Coherent Online Video Style Transfer
• Exploring the Neural Algorithm of Artistic Style
• High-Resolution Image Inpainting using Multi-Scale Neural Patch Synthesis
• ArtFlow: Unbiased Image Style Transfer via Reversible Neural Flows
• Stereoscopic Neural Style Transfer
• DeepMovie: Using Optical Flow and Deep Neural Networks to Stylize Movies
• Deep Feature Consistent Variational Autoencoder
• Controlling Perceptual Factors in Neural Style Transfer
• Learning from Multi-domain Artistic Images for Arbitrary Style Transfer
• Language-Driven Image Style Transfer
• Geometry-Aware Face Completion and Editing
• Computing the Spatial Probability of Inclusion inside Partial Contours for Computer Vision Applications
• Authoring image decompositions with generative models