Deep Learning for Robust Motion Segmentation with Non-Static Cameras

Abstract

This work proposes a new end-to-end DCNN based approach for motion segmentation, especially for video sequences captured with such non-static cameras, called MOSNET. While other approaches focus on spatial or temporal context only, the proposed approach uses 3D convolutions as a key technology to factor in, spatio-temporal features in cohesive video frames. This is done by capturing temporal information in features with a low and also with a high level of abstraction. The lean network architecture with about 21k trainable parameters is mainly based on a pre-trained VGG-16 network. The MOSNET uses a new feature map fusion technique, which enables the network to focus on the appropriate level of abstraction, resolution, and the appropriate size of the receptive field regarding the input. Furthermore, the end-to-end deep learning based approach can be extended by feature based image alignment as a pre-processing step, which brings a gain in performance for some scenes. Evaluating the end-to-end deep learning based MOSNET network in a scene independent manner leads to an overall F-measure of 0.803 on the CDNet2014 dataset. A small temporal window of five input frames, without the need of any initialization is used to obtain this result. Therefore the network is able to perform well on scenes captured with non-static cameras where the image content changes significantly during the scene. In order to get robust results in scenes captured with a moving camera, feature based image alignment can implemented as pre-processing step. The MOSNET combined with pre-processing leads to an F-measure of 0.685 when cross-evaluating with a relabeled LASIESTA dataset, which underpins the capability generalise of the MOSNET.

References in corpus (11)

• Two-Stream Convolutional Networks for Action Recognition in Videos
• An overview of gradient descent optimization algorithms
• A guide to convolution arithmetic for deep learning
• Understanding the Effective Receptive Field in Deep Convolutional Neural Networks
• Object Detectors Emerge in Deep Scene CNNs
• Foreground Segmentation Using a Triplet Convolutional Neural Network for Multiscale Feature Encoding
• YouTube-VOS: A Large-Scale Video Object Segmentation Benchmark
• A Survey on Deep Learning-based Architectures for Semantic Segmentation on 2D images
• What are the Receptive, Effective Receptive, and Projective Fields of Neurons in Convolutional Neural Networks?
• 3DFR: A Swift 3D Feature Reductionist Framework for Scene Independent Change Detection
• Fast Optical Flow using Dense Inverse Search