Recurrent Flow-Guided Semantic Forecasting

Abstract

Understanding the world around us and making decisions about the future is a critical component to human intelligence. As autonomous systems continue to develop, their ability to reason about the future will be the key to their success. Semantic anticipation is a relatively under-explored area for which autonomous vehicles could take advantage of (e.g., forecasting pedestrian trajectories). Motivated by the need for real-time prediction in autonomous systems, we propose to decompose the challenging semantic forecasting task into two subtasks: current frame segmentation and future optical flow prediction. Through this decomposition, we built an efficient, effective, low overhead model with three main components: flow prediction network, feature-flow aggregation LSTM, and end-to-end learnable warp layer. Our proposed method achieves state-of-the-art accuracy on short-term and moving objects semantic forecasting while simultaneously reducing model parameters by up to 95% and increasing efficiency by greater than 40x.

arXiv comment10 pages, 5 figures, 8 tables, Accepted to WACV 2019

References in corpus (22)

• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Caffe: Convolutional Architecture for Fast Feature Embedding
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• Generating Videos with Scene Dynamics
• Decomposing Motion and Content for Natural Video Sequence Prediction
• Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
• Video (language) modeling: a baseline for generative models of natural videos
• Unsupervised Learning for Physical Interaction through Video Prediction
• Spatio-temporal video autoencoder with differentiable memory
• Unsupervised Learning of Disentangled Representations from Video
• Learning to Generate Long-term Future via Hierarchical Prediction
• Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
• Flow-Guided Feature Aggregation for Video Object Detection
• Illuminating Pedestrians via Simultaneous Detection & Segmentation
• Long-term Planning by Short-term Prediction
• On the Sample Complexity of End-to-end Training vs. Semantic Abstraction Training
• Predicting Deeper into the Future of Semantic Segmentation
• Unsupervised Learning of Long-Term Motion Dynamics for Videos
• Pose-Invariant Face Alignment with a Single CNN
• Nonlinear 3D Face Morphable Model
• Video Ladder Networks
• Forecasting Human Dynamics from Static Images