The Unreasonable Effectiveness of Deep Features as a Perceptual Metric

Abstract

While it is nearly effortless for humans to quickly assess the perceptual similarity between two images, the underlying processes are thought to be quite complex. Despite this, the most widely used perceptual metrics today, such as PSNR and SSIM, are simple, shallow functions, and fail to account for many nuances of human perception. Recently, the deep learning community has found that features of the VGG network trained on ImageNet classification has been remarkably useful as a training loss for image synthesis. But how perceptual are these so-called "perceptual losses"? What elements are critical for their success? To answer these questions, we introduce a new dataset of human perceptual similarity judgments. We systematically evaluate deep features across different architectures and tasks and compare them with classic metrics. We find that deep features outperform all previous metrics by large margins on our dataset. More surprisingly, this result is not restricted to ImageNet-trained VGG features, but holds across different deep architectures and levels of supervision (supervised, self-supervised, or even unsupervised). Our results suggest that perceptual similarity is an emergent property shared across deep visual representations.

arXiv commentAccepted to CVPR 2018; Code and data available at https://www.github.com/richzhang/PerceptualSimilarity

References in corpus (10)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
• One weird trick for parallelizing convolutional neural networks
• Unsupervised Visual Representation Learning by Context Prediction
• Unsupervised Learning of Visual Representations using Videos
• Data-dependent Initializations of Convolutional Neural Networks
• Learning to See by Moving
• Deep Networks for Image Super-Resolution with Sparse Prior
• Learning visual groups from co-occurrences in space and time
• Video Frame Interpolation via Adaptive Separable Convolution

Cited by in corpus (139)

• Toward Multimodal Image-to-Image Translation
• NeRF++: Analyzing and Improving Neural Radiance Fields
• Visual Reinforcement Learning with Imagined Goals
• Natural Adversarial Examples
• WaNet -- Imperceptible Warping-based Backdoor Attack
• SPG-Net: Segmentation Prediction and Guidance Network for Image Inpainting
• Deep Nearest Neighbor Anomaly Detection
• Fixing Data Augmentation to Improve Adversarial Robustness
• Convolutional Tensor-Train LSTM for Spatio-temporal Learning
• Controlling generative models with continuous factors of variations
• LDMVFI: Video Frame Interpolation with Latent Diffusion Models
• pixelNeRF: Neural Radiance Fields from One or Few Images
• NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild
• Learnable Gated Temporal Shift Module for Deep Video Inpainting
• ST-MFNet: A Spatio-Temporal Multi-Flow Network for Frame Interpolation
• ShaRF: Shape-conditioned Radiance Fields from a Single View
• On the "steerability" of generative adversarial networks
• Quantifying Perceptual Distortion of Adversarial Examples
• Learning Adaptive Sampling and Reconstruction for Volume Visualization
• VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation
• FitVid: Overfitting in Pixel-Level Video Prediction
• Perspective Projection-Based 3D CT Reconstruction from Biplanar X-rays
• Interactive Image Generation Using Scene Graphs
• Non-Homogeneous Haze Removal via Artificial Scene Prior and Bidimensional Graph Reasoning
• IBRNet: Learning Multi-View Image-Based Rendering
• Long-Horizon Visual Planning with Goal-Conditioned Hierarchical Predictors
• Disentangling Image Distortions in Deep Feature Space
• Depth-aware Blending of Smoothed Images for Bokeh Effect Generation
• Image Inpainting using Block-wise Procedural Training with Annealed Adversarial Counterpart
• One-Shot Free-View Neural Talking-Head Synthesis for Video Conferencing
• DeepView: View Synthesis with Learned Gradient Descent
• Towards Ghost-free Shadow Removal via Dual Hierarchical Aggregation Network and Shadow Matting GAN
• Taming Visually Guided Sound Generation
• 3D Photography using Context-aware Layered Depth Inpainting
• The Effectiveness of Instance Normalization: a Strong Baseline for Single Image Dehazing
• Style Generator Inversion for Image Enhancement and Animation
• Learning Digital Camera Pipeline for Extreme Low-Light Imaging
• Zero-Shot Text-Guided Object Generation with Dream Fields
• A Non-Parametric Test to Detect Data-Copying in Generative Models
• Recover and Identify: A Generative Dual Model for Cross-Resolution Person Re-Identification
• Data Hiding with Deep Learning: A Survey Unifying Digital Watermarking and Steganography
• Single-Image HDR Reconstruction by Learning to Reverse the Camera Pipeline
• Scaling Autoregressive Video Models
• DOVE: Learning Deformable 3D Objects by Watching Videos
• SANNS: Scaling Up Secure Approximate k-Nearest Neighbors Search
• CSGAN: Cyclic-Synthesized Generative Adversarial Networks for Image-to-Image Transformation
• MISO: Mutual Information Loss with Stochastic Style Representations for Multimodal Image-to-Image Translation
• Explaining in Style: Training a GAN to explain a classifier in StyleSpace
• TransGaGa: Geometry-Aware Unsupervised Image-to-Image Translation
• Learning Long-Term Style-Preserving Blind Video Temporal Consistency
• On Interaction Between Augmentations and Corruptions in Natural Corruption Robustness
• DIB-R++: Learning to Predict Lighting and Material with a Hybrid Differentiable Renderer
• Demystifying Inter-Class Disentanglement
• HifiFace: 3D Shape and Semantic Prior Guided High Fidelity Face Swapping
• Positional Normalization
• MCL-GAN: Generative Adversarial Networks with Multiple Specialized Discriminators
• Self-Supervised Shadow Removal
• Performing Video Frame Prediction of Microbial Growth with a Recurrent Neural Network
• Describe What to Change: A Text-guided Unsupervised Image-to-Image Translation Approach
• ReLLIE: Deep Reinforcement Learning for Customized Low-Light Image Enhancement
• Controllable Person Image Synthesis with Spatially-Adaptive Warped Normalization
• Symmetric-Constrained Irregular Structure Inpainting for Brain MRI Registration with Tumor Pathology
• PerceptNet: Learning Perceptual Similarity of Haptic Textures in Presence of Unorderable Triplets
• Revisiting Image Aesthetic Assessment via Self-Supervised Feature Learning
• Not just a matter of semantics: the relationship between visual similarity and semantic similarity
• VORNet: Spatio-temporally Consistent Video Inpainting for Object Removal
• Pose-Guided High-Resolution Appearance Transfer via Progressive Training
• SDIT: Scalable and Diverse Cross-domain Image Translation
• Generative Adversarial Frontal View to Bird View Synthesis
• A Compression Objective and a Cycle Loss for Neural Image Compression
• Following Instructions by Imagining and Reaching Visual Goals
• IMAGINE: Image Synthesis by Image-Guided Model Inversion
• Spatial-Temporal Super-Resolution of Satellite Imagery via Conditional Pixel Synthesis
• Compositional Video Synthesis with Action Graphs
• Dual Attention GANs for Semantic Image Synthesis
• Improving Robustness using Generated Data
• MUST-GAN: Multi-level Statistics Transfer for Self-driven Person Image Generation
• Towards Rolling Shutter Correction and Deblurring in Dynamic Scenes
• Perceptual Video Super Resolution with Enhanced Temporal Consistency
• Direct May Not Be the Best: An Incremental Evolution View of Pose Generation
• Fully Automatic Video Colorization with Self-Regularization and Diversity
• MetaHDR: Model-Agnostic Meta-Learning for HDR Image Reconstruction
• F2GAN: Fusing-and-Filling GAN for Few-shot Image Generation
• Domain-Specific Mappings for Generative Adversarial Style Transfer
• A Self-supervised Approach for Adversarial Robustness
• Defending Against Image Corruptions Through Adversarial Augmentations
• Image-to-Image Translation with Low Resolution Conditioning
• GAN-Based Facial Attractiveness Enhancement
• Face Reconstruction with Variational Autoencoder and Face Masks
• Enforcing Perceptual Consistency on Generative Adversarial Networks by Using the Normalised Laplacian Pyramid Distance
• YASENN: Explaining Neural Networks via Partitioning Activation Sequences
• Example-Based Concept Analysis Framework for Deep Weather Forecast Models
• A Generative Model for Hallucinating Diverse Versions of Super Resolution Images
• Dual In-painting Model for Unsupervised Gaze Correction and Animation in the Wild
• 3D Human Texture Estimation from a Single Image with Transformers
• Learning Symmetry Consistent Deep CNNs for Face Completion
• Semantic Image Manipulation Using Scene Graphs
• Diverse Image Generation via Self-Conditioned GANs
• Deep Consensus Learning
• IEA: Inner Ensemble Average within a convolutional neural network
• Efficient Neural Radiance Fields for Interactive Free-viewpoint Video
• Semantic Text-to-Face GAN -ST^2FG
• One-Shot Image-to-Image Translation via Part-Global Learning with a Multi-adversarial Framework
• Seeing through a Black Box: Toward High-Quality Terahertz TomographicImaging via Multi-Scale Spatio-Spectral Image Fusion
• Embedding Novel Views in a Single JPEG Image
• Introducing Pose Consistency and Warp-Alignment for Self-Supervised 6D Object Pose Estimation in Color Images
• Conditional Adversarial Camera Model Anonymization
• Saliency Driven Perceptual Image Compression
• Lagrangian Objective Function Leads to Improved Unforeseen Attack Generalization in Adversarial Training
• Memory-guided Unsupervised Image-to-image Translation
• Multi-scale Self-calibrated Network for Image Light Source Transfer
• Future Video Synthesis with Object Motion Prediction
• The Unreasonable Effectiveness of Texture Transfer for Single Image Super-resolution
• High-Level Perceptual Similarity is Enabled by Learning Diverse Tasks
• Separating Content and Style for Unsupervised Image-to-Image Translation
• Delving into Rectifiers in Style-Based Image Translation
• Learning Realistic Human Reposing using Cyclic Self-Supervision with 3D Shape, Pose, and Appearance Consistency
• Bridging the Gap between Label- and Reference-based Synthesis in Multi-attribute Image-to-Image Translation
• Sparse to Dense Motion Transfer for Face Image Animation
• Human Imperceptible Attacks and Applications to Improve Fairness
• Quality-aware Unpaired Image-to-Image Translation
• Instance-Conditioned GAN
• Learning Similarity Metrics for Volumetric Simulations with Multiscale CNNs
• Stereo Waterdrop Removal with Row-wise Dilated Attention
• TUCaN: Progressively Teaching Colourisation to Capsules
• MCMI: Multi-Cycle Image Translation with Mutual Information Constraints
• A Shape-Aware Retargeting Approach to Transfer Human Motion and Appearance in Monocular Videos
• Video Super-Resolution with Long-Term Self-Exemplars
• Localized Uncertainty Attacks
• Unsupervised Video Prediction from a Single Frame by Estimating 3D Dynamic Scene Structure
• Collaboration among Image and Object Level Features for Image Colourisation
• Learning Structral coherence Via Generative Adversarial Network for Single Image Super-Resolution
• Coarse-to-Fine Gaze Redirection with Numerical and Pictorial Guidance
• Analysis of Macula on Color Fundus Images Using Heightmap Reconstruction Through Deep Learning
• Hierarchical Modes Exploring in Generative Adversarial Networks
• Synthesizing Photorealistic Images with Deep Generative Learning
• ReshapeGAN: Object Reshaping by Providing A Single Reference Image
• Shared Visual Representations of Drawing for Communication: How do different biases affect human interpretability and intent?
• Seeing Eye to AI? Applying Deep-Feature-Based Similarity Metrics to Information Visualization