On the Transfer of Inductive Bias from Simulation to the Real World: a New Disentanglement Dataset

Abstract

Learning meaningful and compact representations with disentangled semantic aspects is considered to be of key importance in representation learning. Since real-world data is notoriously costly to collect, many recent state-of-the-art disentanglement models have heavily relied on synthetic toy data-sets. In this paper, we propose a novel data-set which consists of over one million images of physical 3D objects with seven factors of variation, such as object color, shape, size and position. In order to be able to control all the factors of variation precisely, we built an experimental platform where the objects are being moved by a robotic arm. In addition, we provide two more datasets which consist of simulations of the experimental setup. These datasets provide for the first time the possibility to systematically investigate how well different disentanglement methods perform on real data in comparison to simulation, and how simulated data can be leveraged to build better representations of the real world. We provide a first experimental study of these questions and our results indicate that learned models transfer poorly, but that model and hyperparameter selection is an effective means of transferring information to the real world.

arXiv commentNeurIPS 2019 Camera Ready Version

Cited by in corpus (19)

• Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style
• Disentangling Factors of Variation Using Few Labels
• Fair Representation: Guaranteeing Approximate Multiple Group Fairness for Unknown Tasks
• Towards Nonlinear Disentanglement in Natural Data with Temporal Sparse Coding
• 3D-IDS: Doubly Disentangled Dynamic Intrusion Detection
• Semi-Supervised StyleGAN for Disentanglement Learning
• On the Transfer of Disentangled Representations in Realistic Settings
• On the Fairness of Disentangled Representations
• Visual Representation Learning Does Not Generalize Strongly Within the Same Domain
• Generalization and Robustness Implications in Object-Centric Learning
• A Fine-Grained Analysis on Distribution Shift
• DCI-ES: An Extended Disentanglement Framework with Connections to Identifiability
• Deep Anomaly Detection by Residual Adaptation
• Disentanglement Learning via Topology
• Causality in Neural Networks -- An Extended Abstract
• X-model: Improving Data Efficiency in Deep Learning with A Minimax Model
• NeurIPS 2019 Disentanglement Challenge: Improved Disentanglement through Learned Aggregation of Convolutional Feature Maps
• Local Disentanglement in Variational Auto-Encoders Using Jacobian $L_1$ Regularization
• DAReN: A Collaborative Approach Towards Reasoning And Disentangling