Domain Decluttering: Simplifying Images to Mitigate Synthetic-Real Domain Shift and Improve Depth Estimation

Abstract

Leveraging synthetically rendered data offers great potential to improve monocular depth estimation and other geometric estimation tasks, but closing the synthetic-real domain gap is a non-trivial and important task. While much recent work has focused on unsupervised domain adaptation, we consider a more realistic scenario where a large amount of synthetic training data is supplemented by a small set of real images with ground-truth. In this setting, we find that existing domain translation approaches are difficult to train and offer little advantage over simple baselines that use a mix of real and synthetic data. A key failure mode is that real-world images contain novel objects and clutter not present in synthetic training. This high-level domain shift isn't handled by existing image translation models. Based on these observations, we develop an attention module that learns to identify and remove difficult out-of-domain regions in real images in order to improve depth prediction for a model trained primarily on synthetic data. We carry out extensive experiments to validate our attend-remove-complete approach (ARC) and find that it significantly outperforms state-of-the-art domain adaptation methods for depth prediction. Visualizing the removed regions provides interpretable insights into the synthetic-real domain gap.

arXiv commentcamera-ready version, CVPR2020

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Overcoming catastrophic forgetting in neural networks
• Categorical Reparameterization with Gumbel-Softmax
• Learning Transferable Features with Deep Adaptation Networks
• Convolutional Sequence to Sequence Learning
• Deep Domain Confusion: Maximizing for Domain Invariance
• Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Toward Multimodal Image-to-Image Translation
• Learning from Synthetic Humans
• DropBlock: A regularization method for convolutional networks
• Adversarial Scene Editing: Automatic Object Removal from Weak Supervision