STFCN: Spatio-Temporal FCN for Semantic Video Segmentation

Abstract

This paper presents a novel method to involve both spatial and temporal features for semantic video segmentation. Current work on convolutional neural networks(CNNs) has shown that CNNs provide advanced spatial features supporting a very good performance of solutions for both image and video analysis, especially for the semantic segmentation task. We investigate how involving temporal features also has a good effect on segmenting video data. We propose a module based on a long short-term memory (LSTM) architecture of a recurrent neural network for interpreting the temporal characteristics of video frames over time. Our system takes as input frames of a video and produces a correspondingly-sized output; for segmenting the video our method combines the use of three components: First, the regional spatial features of frames are extracted using a CNN; then, using LSTM the temporal features are added; finally, by deconvolving the spatio-temporal features we produce pixel-wise predictions. Our key insight is to build spatio-temporal convolutional networks (spatio-temporal CNNs) that have an end-to-end architecture for semantic video segmentation. We adapted fully some known convolutional network architectures (such as FCN-AlexNet and FCN-VGG16), and dilated convolution into our spatio-temporal CNNs. Our spatio-temporal CNNs achieve state-of-the-art semantic segmentation, as demonstrated for the Camvid and NYUDv2 datasets.

References in corpus (7)

• Two-Stream Convolutional Networks for Action Recognition in Videos
• FlowNet: Learning Optical Flow with Convolutional Networks
• Grid Long Short-Term Memory
• Decoupled Deep Neural Network for Semi-supervised Semantic Segmentation
• Attention to Scale: Scale-aware Semantic Image Segmentation
• Real-Time Anomaly Detection and Localization in Crowded Scenes
• Pixel-wise Segmentation of Street with Neural Networks

Cited by in corpus (12)

• Flow-Guided Feature Aggregation for Video Object Detection
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
• On The Stability of Video Detection and Tracking
• Semantic Video Segmentation: A Review on Recent Approaches
• Unsupervised Domain Adaptation for Video Semantic Segmentation
• Low-Latency Video Semantic Segmentation
• Automatic Real-time Background Cut for Portrait Videos
• Every Frame Counts: Joint Learning of Video Segmentation and Optical Flow
• Functionally Modular and Interpretable Temporal Filtering for Robust Segmentation
• Beyond Single Stage Encoder-Decoder Networks: Deep Decoders for Semantic Image Segmentation
• Tom: Leveraging trend of the observed gradients for faster convergence
• Veritatem Dies Aperit- Temporally Consistent Depth Prediction Enabled by a Multi-Task Geometric and Semantic Scene Understanding Approach