E-LPIPS: Robust Perceptual Image Similarity via Random Transformation Ensembles

Abstract

It has been recently shown that the hidden variables of convolutional neural networks make for an efficient perceptual similarity metric that accurately predicts human judgment on relative image similarity assessment. First, we show that such learned perceptual similarity metrics (LPIPS) are susceptible to adversarial attacks that dramatically contradict human visual similarity judgment. While this is not surprising in light of neural networks' well-known weakness to adversarial perturbations, we proceed to show that self-ensembling with an infinite family of random transformations of the input --- a technique known not to render classification networks robust --- is enough to turn the metric robust against attack, while retaining predictive power on human judgments. Finally, we study the geometry imposed by our our novel self-ensembled metric (E-LPIPS) on the space of natural images. We find evidence of "perceptual convexity" by showing that convex combinations of similar-looking images retain appearance, and that discrete geodesics yield meaningful frame interpolation and texture morphing, all without explicit correspondences.

arXiv commentCode and supplemental material available at https://github.com/mkettune/elpips/

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• One weird trick for parallelizing convolutional neural networks
• The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale
• Mitigating Adversarial Effects Through Randomization
• Accurate Image Super-Resolution Using Very Deep Convolutional Networks
• Eigen-Distortions of Hierarchical Representations

Cited by in corpus (8)

• Smooth Adversarial Training
• FreeStyleGAN: Free-view Editable Portrait Rendering with the Camera Manifold
• A Loss Function for Generative Neural Networks Based on Watson's Perceptual Model
• Improving Robustness of Reinforcement Learning for Power System Control with Adversarial Training
• Light Stage Super-Resolution: Continuous High-Frequency Relighting
• Stateful Detection of Black-Box Adversarial Attacks
• Deep scene-scale material estimation from multi-view indoor captures
• Lagrangian Objective Function Leads to Improved Unforeseen Attack Generalization in Adversarial Training