Perceptual Losses for Real-Time Style Transfer and Super-Resolution

Abstract

We consider image transformation problems, where an input image is transformed into an output image. Recent methods for such problems typically train feed-forward convolutional neural networks using a \emph{per-pixel} loss between the output and ground-truth images. Parallel work has shown that high-quality images can be generated by defining and optimizing \emph{perceptual} loss functions based on high-level features extracted from pretrained networks. We combine the benefits of both approaches, and propose the use of perceptual loss functions for training feed-forward networks for image transformation tasks. We show results on image style transfer, where a feed-forward network is trained to solve the optimization problem proposed by Gatys et al in real-time. Compared to the optimization-based method, our network gives similar qualitative results but is three orders of magnitude faster. We also experiment with single-image super-resolution, where replacing a per-pixel loss with a perceptual loss gives visually pleasing results.

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• Understanding Neural Networks Through Deep Visualization
• cuDNN: Efficient Primitives for Deep Learning
• Learning Deconvolution Network for Semantic Segmentation
• Texture Synthesis Using Convolutional Neural Networks
• Image Super-Resolution Using Deep Convolutional Networks
• Recurrent Convolutional Neural Networks for Scene Parsing

Cited by in corpus (76)

• Low Dose CT Image Denoising Using a Generative Adversarial Network with Wasserstein Distance and Perceptual Loss
• Vision Transformers for Single Image Dehazing
• Image De-raining Using a Conditional Generative Adversarial Network
• Harnessing Deep Neural Networks with Logic Rules
• Deep Identity-aware Transfer of Facial Attributes
• Deep Generative Adversarial Networks for Compressed Sensing Automates MRI
• Structure-Preserving Image Super-Resolution
• Is the deconvolution layer the same as a convolutional layer?
• Face Generation and Editing with StyleGAN: A Survey
• Deep Reinforcement Learning for Visual Object Tracking in Videos
• CPP-Net: Context-aware Polygon Proposal Network for Nucleus Segmentation
• Controlling generative models with continuous factors of variations
• StyleBank: An Explicit Representation for Neural Image Style Transfer
• Image Quality Assessment for Magnetic Resonance Imaging
• Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks
• A Practical Contrastive Learning Framework for Single-Image Super-Resolution
• Robust Differentiable SVD
• HybrUR: A Hybrid Physical-Neural Solution for Unsupervised Underwater Image Restoration
• Making DeepFakes more spurious: evading deep face forgery detection via trace removal attack
• MWFormer: Multi-Weather Image Restoration Using Degradation-Aware Transformers
• Toward Real-World Super-Resolution via Adaptive Downsampling Models
• Hyperspectral Pansharpening: Critical Review, Tools and Future Perspectives
• Convolutional Network for Attribute-driven and Identity-preserving Human Face Generation
• SceneDreamer: Unbounded 3D Scene Generation from 2D Image Collections
• Coherent Online Video Style Transfer
• Recurrent Generative Adversarial Networks for Proximal Learning and Automated Compressive Image Recovery
• RSHazeDiff: A Unified Fourier-aware Diffusion Model for Remote Sensing Image Dehazing
• Convolutional Neural Networks Analyzed via Convolutional Sparse Coding
• Adversarial Sparse-View CBCT Artifact Reduction
• PreCNet: Next-Frame Video Prediction Based on Predictive Coding
• Quad-Net: Quad-domain Network for CT Metal Artifact Reduction
• Gap-closing Matters: Perceptual Quality Evaluation and Optimization of Low-Light Image Enhancement
• Semi-DerainGAN: A New Semi-supervised Single Image Deraining Network
• DFR-Net: Density Feature Refinement Network for Image Dehazing Utilizing Haze Density Difference
• Uncovering the Over-smoothing Challenge in Image Super-Resolution: Entropy-based Quantification and Contrastive Optimization
• Benchmarking and Improving Bird's Eye View Perception Robustness in Autonomous Driving
• Stereoscopic Neural Style Transfer
• Semi-Supervised Coupled Thin-Plate Spline Model for Rotation Correction and Beyond
• A Specific Task-oriented Semantic Image Communication System for substation patrol inspection
• SARGAN: Spatial Attention-based Residuals for Facial Expression Manipulation
• IFQA: Interpretable Face Quality Assessment
• A Coarse-to-Fine Multi-stream Hybrid Deraining Network for Single Image Deraining
• Deep Feature Consistent Variational Autoencoder
• Passive Non-Line-of-Sight Imaging with Light Transport Modulation
• Treatment Learning Causal Transformer for Noisy Image Classification
• M-to-N Backdoor Paradigm: A Multi-Trigger and Multi-Target Attack to Deep Learning Models
• Few-shot Image Generation via Masked Discrimination
• Learning Residual Images for Face Attribute Manipulation
• FastCLIPstyler: Optimisation-free Text-based Image Style Transfer Using Style Representations
• Memories are One-to-Many Mapping Alleviators in Talking Face Generation
• Retargeting video with an end-to-end framework
• Machine-learned models for magnetic materials
• One-to-Many Network for Visually Pleasing Compression Artifacts Reduction
• BS-LDM: Effective Bone Suppression in High-Resolution Chest X-Ray Images with Conditional Latent Diffusion Models
• A Machine Learning Approach to Correcting Atmospheric Seeing in Solar Flare Observations
• Semantic Image Synthesis via Class-Adaptive Cross-Attention
• Transposer: Universal Texture Synthesis Using Feature Maps as Transposed Convolution Filter
• Learning Parallax for Stereo Event-based Motion Deblurring
• Human perception in computer vision
• MR to X-Ray Projection Image Synthesis
• Generalized Deep Image to Image Regression
• Multi-scale Restoration of Missing Data in Optical Time-series Images with Masked Spatial-Temporal Attention Network
• Towards Label-Free 3D Segmentation of Optical Coherence Tomography Images of the Optic Nerve Head Using Deep Learning
• MM-NeRF: Multimodal-Guided 3D Multi-Style Transfer of Neural Radiance Field
• Domain-Specific Mappings for Generative Adversarial Style Transfer
• Universal Vessel Segmentation for Multi-Modality Retinal Images
• DeMeshNet: Blind Face Inpainting for Deep MeshFace Verification
• Learning Motion Flows for Semi-supervised Instrument Segmentation from Robotic Surgical Video
• Chinese/English mixed Character Segmentation as Semantic Segmentation
• An End-to-End Depth-Based Pipeline for Selfie Image Rectification
• A Proof of Concept Resource Management Scheme for Augmented Reality Applications in 5G Systems
• Trade-offs in Privacy-Preserving Eye Tracking through Iris Obfuscation: A Benchmarking Study
• Eliminating the Blind Spot: Adapting 3D Object Detection and Monocular Depth Estimation to 360° Panoramic Imagery
• DOOMGAN:High-Fidelity Dynamic Identity Obfuscation Ocular Generative Morphing
• D-LORD for Motion Stylization
• Leveraging GANs to Improve Continuous Path Keyboard Input Models