Texture Synthesis Using Convolutional Neural Networks

Abstract

Here we introduce a new model of natural textures based on the feature spaces of convolutional neural networks optimised for object recognition. Samples from the model are of high perceptual quality demonstrating the generative power of neural networks trained in a purely discriminative fashion. Within the model, textures are represented by the correlations between feature maps in several layers of the network. We show that across layers the texture representations increasingly capture the statistical properties of natural images while making object information more and more explicit. The model provides a new tool to generate stimuli for neuroscience and might offer insights into the deep representations learned by convolutional neural networks.

Revision for NIPS 2015 Camera Ready. In line with reviewer's comments we now focus on the texture model and texture synthesis performance. We limit the relationship of our texture model to the ventral stream and its potential use for neuroscience to the discussion of the paper

References in corpus (6)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• Deep convolutional filter banks for texture recognition and segmentation

Cited by in corpus (100)

• ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks
• Perceptual Losses for Real-Time Style Transfer and Super-Resolution
• Density estimation using Real NVP
• Texture Networks: Feed-forward Synthesis of Textures and Stylized Images
• Visualizing Deep Convolutional Neural Networks Using Natural Pre-Images
• A note on the evaluation of generative models
• Microstructural Materials Design via Deep Adversarial Learning Methodology
• Stable and Controllable Neural Texture Synthesis and Style Transfer Using Histogram Losses
• Non-Stationary Texture Synthesis by Adversarial Expansion
• Artistic style transfer for videos and spherical images
• Multi-style Generative Network for Real-time Transfer
• Texture Synthesis with Spatial Generative Adversarial Networks
• Wavelet Convolutional Neural Networks for Texture Classification
• Attribute2Image: Conditional Image Generation from Visual Attributes
• Super-Resolution with Deep Convolutional Sufficient Statistics
• StyleBank: An Explicit Representation for Neural Image Style Transfer
• PhotoShape: Photorealistic Materials for Large-Scale Shape Collections
• Fully Convolutional Adaptation Networks for Semantic Segmentation
• TileGAN: Synthesis of Large-Scale Non-Homogeneous Textures
• Frustratingly Simple Domain Generalization via Image Stylization
• A Closed-form Solution to Photorealistic Image Stylization
• How convolutional neural network see the world - A survey of convolutional neural network visualization methods
• ST-MFNet: A Spatio-Temporal Multi-Flow Network for Frame Interpolation
• Stacked Generative Adversarial Networks
• Image Companding and Inverse Halftoning using Deep Convolutional Neural Networks
• Hallucinated-IQA: No-Reference Image Quality Assessment via Adversarial Learning
• Structural inpainting
• Texture Synthesis Using Shallow Convolutional Networks with Random Filters
• Visual Attribute Transfer through Deep Image Analogy
• InGAN: Capturing and Remapping the "DNA" of a Natural Image
• A Systematic Survey of Deep Learning-based Single-Image Super-Resolution
• Single Image Reflection Removal Using Deep Encoder-Decoder Network
• Perceptual Image Anomaly Detection
• Exploring the Neural Algorithm of Artistic Style
• Diversified Texture Synthesis with Feed-forward Networks
• Texture Synthesis Through Convolutional Neural Networks and Spectrum Constraints
• TextureGAN: Controlling Deep Image Synthesis with Texture Patches
• Style Mixer: Semantic-aware Multi-Style Transfer Network
• Incorporating long-range consistency in CNN-based texture generation
• Content Aware Neural Style Transfer
• Fashioning with Networks: Neural Style Transfer to Design Clothes
• Applying Visual Domain Style Transfer and Texture Synthesis Techniques to Audio - Insights and Challenges
• Zero-Shot Visual Recognition using Semantics-Preserving Adversarial Embedding Networks
• Characterizing and Improving Stability in Neural Style Transfer
• Dynamic Instance Normalization for Arbitrary Style Transfer
• Hybrid Loss for Learning Single-Image-based HDR Reconstruction
• Towards Metamerism via Foveated Style Transfer
• Stroke Controllable Fast Style Transfer with Adaptive Receptive Fields
• Structured Prediction Energy Networks
• GramGAN: Deep 3D Texture Synthesis From 2D Exemplars
• A Novel Convolutional Neural Network for Image Steganalysis with Shared Normalization
• Deep Feature Consistent Variational Autoencoder
• Avatar-Net: Multi-scale Zero-shot Style Transfer by Feature Decoration
• Learning to Generate Time-Lapse Videos Using Multi-Stage Dynamic Generative Adversarial Networks
• Synthesising Dynamic Textures using Convolutional Neural Networks
• A Style-Aware Content Loss for Real-time HD Style Transfer
• Images Don't Lie: Transferring Deep Visual Semantic Features to Large-Scale Multimodal Learning to Rank
• Controlling Perceptual Factors in Neural Style Transfer
• Adversarial nets with perceptual losses for text-to-image synthesis
• Deep Semantic Face Deblurring
• DeepWarp: Photorealistic Image Resynthesis for Gaze Manipulation
• SideEye: A Generative Neural Network Based Simulator of Human Peripheral Vision
• Multimodal Transfer: A Hierarchical Deep Convolutional Neural Network for Fast Artistic Style Transfer
• Rethinking and Improving the Robustness of Image Style Transfer
• Learning from Multi-domain Artistic Images for Arbitrary Style Transfer
• Image Enhanced Rotation Prediction for Self-Supervised Learning
• NaturalInversion: Data-Free Image Synthesis Improving Real-World Consistency
• Maximum Entropy Flow Networks
• Reference-Conditioned Super-Resolution by Neural Texture Transfer
• WarpGAN: Automatic Caricature Generation
• Learning to Synthesize Fashion Textures
• Deep Learning for Explicitly Modeling Optimization Landscapes
• Synthesizing Diverse, High-Quality Audio Textures
• Improved Texture Networks: Maximizing Quality and Diversity in Feed-forward Stylization and Texture Synthesis
• Domain-Specific Mappings for Generative Adversarial Style Transfer
• Cross-convolutional-layer Pooling for Image Recognition
• Learning to Caricature via Semantic Shape Transform
• Image Inpainting for High-Resolution Textures using CNN Texture Synthesis
• TET-GAN: Text Effects Transfer via Stylization and Destylization
• A geometrically aware auto-encoder for multi-texture synthesis
• Diverse feature visualizations reveal invariances in early layers of deep neural networks
• Denoising of 3-D Magnetic Resonance Images Using a Residual Encoder-Decoder Wasserstein Generative Adversarial Network
• Texture Synthesis with Recurrent Variational Auto-Encoder
• Awesome Typography: Statistics-Based Text Effects Transfer
• On Random Weights for Texture Generation in One Layer Neural Networks
• Evolvement Constrained Adversarial Learning for Video Style Transfer
• How big can style be? Addressing high dimensionality for recommending with style
• Learning to Look Around: Intelligently Exploring Unseen Environments for Unknown Tasks
• Wasserstein Introspective Neural Networks
• Steganography Security: Principle and Practice
• Content-based similar document image retrieval using fusion of CNN features
• Taming Adversarial Domain Transfer with Structural Constraints for Image Enhancement
• MSCE: An edge preserving robust loss function for improving super-resolution algorithms
• TLGAN: document Text Localization using Generative Adversarial Nets
• Evolution of Images with Diversity and Constraints Using a Generator Network
• Sensory Polymorphism and Behavior: When Machine Vision Meets Monkey Eyes
• A Content Transformation Block For Image Style Transfer
• Authoring image decompositions with generative models
• Patch redundancy in images: a statistical testing framework and some applications
• Emerging Applications of Reversible Data Hiding