Understanding deep features with computer-generated imagery

Abstract

We introduce an approach for analyzing the variation of features generated by convolutional neural networks (CNNs) with respect to scene factors that occur in natural images. Such factors may include object style, 3D viewpoint, color, and scene lighting configuration. Our approach analyzes CNN feature responses corresponding to different scene factors by controlling for them via rendering using a large database of 3D CAD models. The rendered images are presented to a trained CNN and responses for different layers are studied with respect to the input scene factors. We perform a decomposition of the responses based on knowledge of the input scene factors and analyze the resulting components. In particular, we quantify their relative importance in the CNN responses and visualize them using principal component analysis. We show qualitative and quantitative results of our study on three CNNs trained on large image datasets: AlexNet, Places, and Oxford VGG. We observe important differences across the networks and CNN layers for different scene factors and object categories. Finally, we demonstrate that our analysis based on computer-generated imagery translates to the network representation of natural images.

Cited by in corpus (21)

• What makes ImageNet good for transfer learning?
• SceneNet: Understanding Real World Indoor Scenes With Synthetic Data
• i-RevNet: Deep Invertible Networks
• Procedural Modeling and Physically Based Rendering for Synthetic Data Generation in Automotive Applications
• Interpreting CNN Knowledge via an Explanatory Graph
• Facial Landmark Detection with Tweaked Convolutional Neural Networks
• Interpretable Convolutional Neural Networks
• Interactively Transferring CNN Patterns for Part Localization
• Examining CNN Representations with respect to Dataset Bias
• Mining Object Parts from CNNs via Active Question-Answering
• Physically-Based Rendering for Indoor Scene Understanding Using Convolutional Neural Networks
• Building a Regular Decision Boundary with Deep Networks
• Explanatory Graphs for CNNs
• Mining Interpretable AOG Representations from Convolutional Networks via Active Question Answering
• On the effect of pooling on the geometry of representations
• Interpretable Neural Network Decoupling
• Class Subset Selection for Transfer Learning using Submodularity
• On the Computation of Neumann Series
• cvpaper.challenge in 2016: Futuristic Computer Vision through 1,600 Papers Survey
• Understanding learned CNN features through Filter Decoding with Substitution
• From Virtual to Real World Visual Perception using Domain Adaptation -- The DPM as Example