Foreground-Aware Stylization and Consensus Pseudo-Labeling for Domain Adaptation of First-Person Hand Segmentation

Abstract

Hand segmentation is a crucial task in first-person vision. Since first-person images exhibit strong bias in appearance among different environments, adapting a pre-trained segmentation model to a new domain is required in hand segmentation. Here, we focus on appearance gaps for hand regions and backgrounds separately. We propose (i) foreground-aware image stylization and (ii) consensus pseudo-labeling for domain adaptation of hand segmentation. We stylize source images independently for the foreground and background using target images as style. To resolve the domain shift that the stylization has not addressed, we apply careful pseudo-labeling by taking a consensus between the models trained on the source and stylized source images. We validated our method on domain adaptation of hand segmentation from real and simulation images. Our method achieved state-of-the-art performance in both settings. We also demonstrated promising results in challenging multi-target domain adaptation and domain generalization settings. Code is available at https://github.com/ut-vision/FgSty-CPL.

arXiv commentAccepted to IEEE Access 2021

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Unsupervised Domain Adaptation by Backpropagation
• ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation
• Conditional Adversarial Domain Adaptation
• Unsupervised Domain Adaptation with Residual Transfer Networks
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• Asymmetric Tri-training for Unsupervised Domain Adaptation
• The 2018 DAVIS Challenge on Video Object Segmentation
• Analysis of the hands in egocentric vision: A survey
• Privacy-Preserving Human Activity Recognition from Extreme Low Resolution
• Towards Storytelling from Visual Lifelogging: An Overview
• DCAN: Dual Channel-wise Alignment Networks for Unsupervised Scene Adaptation
• Keep it Simple: Image Statistics Matching for Domain Adaptation
• Ego2Hands: A Dataset for Egocentric Two-hand Segmentation and Detection