Self-supervised Video Object Segmentation by Motion Grouping
arXiv:2104.07658
Abstract
Animals have evolved highly functional visual systems to understand motion, assisting perception even under complex environments. In this paper, we work towards developing a computer vision system able to segment objects by exploiting motion cues, i.e. motion segmentation. We make the following contributions: First, we introduce a simple variant of the Transformer to segment optical flow frames into primary objects and the background. Second, we train the architecture in a self-supervised manner, i.e. without using any manual annotations. Third, we analyze several critical components of our method and conduct thorough ablation studies to validate their necessity. Fourth, we evaluate the proposed architecture on public benchmarks (DAVIS2016, SegTrackv2, and FBMS59). Despite using only optical flow as input, our approach achieves superior or comparable results to previous state-of-the-art self-supervised methods, while being an order of magnitude faster. We additionally evaluate on a challenging camouflage dataset (MoCA), significantly outperforming the other self-supervised approaches, and comparing favourably to the top supervised approach, highlighting the importance of motion cues, and the potential bias towards visual appearance in existing video segmentation models.
arXiv commentBest Paper in CVPR2021 RVSU Workshop. Accepted by ICCV
References in corpus (14)
- Instance Normalization: The Missing Ingredient for Fast Stylization
- Is Space-Time Attention All You Need for Video Understanding?
- Zero-Shot Text-to-Image Generation
- YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark
- Self-supervised Co-training for Video Representation Learning
- Multi-Object Representation Learning with Iterative Variational Inference
- MONet: Unsupervised Scene Decomposition and Representation
- FusionSeg: Learning to combine motion and appearance for fully automatic segmention of generic objects in videos
- GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representations
- Sim2real transfer learning for 3D human pose estimation: motion to the rescue
- Self-supervised Learning for Video Correspondence Flow
- A Detailed Rubric for Motion Segmentation
- Anchor Diffusion for Unsupervised Video Object Segmentation
- Making a Case for 3D Convolutions for Object Segmentation in Videos