Understanding Deep Image Representations by Inverting Them

Abstract

Image representations, from SIFT and Bag of Visual Words to Convolutional Neural Networks (CNNs), are a crucial component of almost any image understanding system. Nevertheless, our understanding of them remains limited. In this paper we conduct a direct analysis of the visual information contained in representations by asking the following question: given an encoding of an image, to which extent is it possible to reconstruct the image itself? To answer this question we contribute a general framework to invert representations. We show that this method can invert representations such as HOG and SIFT more accurately than recent alternatives while being applicable to CNNs too. We then use this technique to study the inverse of recent state-of-the-art CNN image representations for the first time. Among our findings, we show that several layers in CNNs retain photographically accurate information about the image, with different degrees of geometric and photometric invariance.

Cited by in corpus (49)

• Semantic Style Transfer and Turning Two-Bit Doodles into Fine Artworks
• Visualizing and Understanding Neural Models in NLP
• Machine Learning of Explicit Order Parameters: From the Ising Model to SU(2) Lattice Gauge Theory
• Amulet: Aggregating Multi-level Convolutional Features for Salient Object Detection
• Multi-style Generative Network for Real-time Transfer
• What Do We Understand About Convolutional Networks?
• Adversarial Manipulation of Deep Representations
• A Taxonomy and Library for Visualizing Learned Features in Convolutional Neural Networks
• AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence
• A deep architecture for unified aesthetic prediction
• Derivatives and Inverse of Cascaded Linear+Nonlinear Neural Models
• Multi-scale Deep Learning Architectures for Person Re-identification
• Towards falsifiable interpretability research
• DeepMiner: Discovering Interpretable Representations for Mammogram Classification and Explanation
• Visual Representations: Defining Properties and Deep Approximations
• Interpretable Convolutional Neural Networks
• Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks
• Evaluating the visualization of what a Deep Neural Network has learned
• Interactively Transferring CNN Patterns for Part Localization
• Borrowing Treasures from the Wealthy: Deep Transfer Learning through Selective Joint Fine-tuning
• Visualizing and Understanding Deep Texture Representations
• Cooperative Learning with Visual Attributes
• LEP-CNN: A Lightweight Edge Device Assisted Privacy-preserving CNN Inference Solution for IoT
• Understanding Individual Decisions of CNNs via Contrastive Backpropagation
• Analyzing Classifiers: Fisher Vectors and Deep Neural Networks
• Right whale recognition using convolutional neural networks
• All-In-One: Facial Expression Transfer, Editing and Recognition Using A Single Network
• Patch-wise++ Perturbation for Adversarial Targeted Attacks
• Mining Object Parts from CNNs via Active Question-Answering
• Egocentric Basketball Motion Planning from a Single First-Person Image
• Top-down Visual Saliency Guided by Captions
• A Unified Framework for Generalizable Style Transfer: Style and Content Separation
• Deep Multi-Modal Image Correspondence Learning
• Unsupervised Regenerative Learning of Hierarchical Features in Spiking Deep Networks for Object Recognition
• Context Augmentation for Convolutional Neural Networks
• Analyzing Stability of Convolutional Neural Networks in the Frequency Domain
• Learning Class Unique Features in Fine-Grained Visual Classification
• Design of Kernels in Convolutional Neural Networks for Image Classification
• Differential Geometric Retrieval of Deep Features
• See the Difference: Direct Pre-Image Reconstruction and Pose Estimation by Differentiating HOG
• Manifestation of Image Contrast in Deep Networks
• Learning Image Matching by Simply Watching Video
• Analyzing Learned Convnet Features with Dirichlet Process Gaussian Mixture Models
• Understanding learned CNN features through Filter Decoding with Substitution
• Train, Diagnose and Fix: Interpretable Approach for Fine-grained Action Recognition
• Constraint-free Natural Image Reconstruction from fMRI Signals Based on Convolutional Neural Network
• Collaborative creativity with Monte-Carlo Tree Search and Convolutional Neural Networks
• Layer Dynamics of Linearised Neural Nets
• Analyzing Representations inside Convolutional Neural Networks