Leveraging Motion Priors in Videos for Improving Human Segmentation

Abstract

Despite many advances in deep-learning based semantic segmentation, performance drop due to distribution mismatch is often encountered in the real world. Recently, a few domain adaptation and active learning approaches have been proposed to mitigate the performance drop. However, very little attention has been made toward leveraging information in videos which are naturally captured in most camera systems. In this work, we propose to leverage "motion prior" in videos for improving human segmentation in a weakly-supervised active learning setting. By extracting motion information using optical flow in videos, we can extract candidate foreground motion segments (referred to as motion prior) potentially corresponding to human segments. We propose to learn a memory-network-based policy model to select strong candidate segments (referred to as strong motion prior) through reinforcement learning. The selected segments have high precision and are directly used to finetune the model. In a newly collected surveillance camera dataset and a publicly available UrbanStreet dataset, our proposed method improves the performance of human segmentation across multiple scenes and modalities (i.e., RGB to Infrared (IR)). Last but not least, our method is empirically complementary to existing domain adaptation approaches such that additional performance gain is achieved by combining our weakly-supervised active learning approach with domain adaptation approaches.

arXiv commentECCV2018

References in corpus (16)

• Learning Transferable Features with Deep Adaptation Networks
• Unsupervised Domain Adaptation by Backpropagation
• FCNs in the Wild: Pixel-level Adversarial and Constraint-based Adaptation
• Domain Separation Networks
• Deep Bayesian Active Learning with Image Data
• Curriculum Domain Adaptation for Semantic Segmentation of Urban Scenes
• Adversarial Discriminative Domain Adaptation
• Central Moment Discrepancy (CMD) for Domain-Invariant Representation Learning
• Adversarial Active Learning for Deep Networks: a Margin Based Approach
• Control of Memory, Active Perception, and Action in Minecraft
• A Variance Maximization Criterion for Active Learning
• Learning Algorithms for Active Learning
• Learning how to Active Learn: A Deep Reinforcement Learning Approach
• On Face Segmentation, Face Swapping, and Face Perception
• Video Human Segmentation using Fuzzy Object Models and its Application to Body Pose Estimation of Toddlers for Behavior Studies
• Motion Segmentation via Global and Local Sparse Subspace Optimization