From Recognition to Prediction: Analysis of Human Action and Trajectory Prediction in Video

Abstract

With the advancement in computer vision deep learning, systems now are able to analyze an unprecedented amount of rich visual information from videos to enable applications such as autonomous driving, socially-aware robot assistant and public safety monitoring. Deciphering human behaviors to predict their future paths/trajectories and what they would do from videos is important in these applications. However, human trajectory prediction still remains a challenging task, as scene semantics and human intent are difficult to model. Many systems do not provide high-level semantic attributes to reason about pedestrian future. This design hinders prediction performance in video data from diverse domains and unseen scenarios. To enable optimal future human behavioral forecasting, it is crucial for the system to be able to detect and analyze human activities as well as scene semantics, passing informative features to the subsequent prediction module for context understanding.

arXiv commentPh.D. Thesis. Version 2: Defense. See here: https://junweiliang.github.io/thesis/

References in corpus (37)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• ADADELTA: An Adaptive Learning Rate Method
• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• R-FCN: Object Detection via Region-based Fully Convolutional Networks
• Unsupervised Learning of Video Representations using LSTMs
• Learning to Reweight Examples for Robust Deep Learning
• Emergence of Locomotion Behaviours in Rich Environments
• Training Convolutional Networks with Noisy Labels
• Exploiting Feature and Class Relationships in Video Categorization with Regularized Deep Neural Networks
• A Short Note on the Kinetics-700-2020 Human Action Dataset
• A Simple, Fast Diverse Decoding Algorithm for Neural Generation
• Overcoming Limitations of Mixture Density Networks: A Sampling and Fitting Framework for Multimodal Future Prediction
• MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Prediction
• Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels
• Multiple Futures Prediction
• Trajectory Forecasts in Unknown Environments Conditioned on Grid-Based Plans
• Scene-LSTM: A Model for Human Trajectory Prediction
• Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videos
• Diverse Trajectory Forecasting with Determinantal Point Processes
• The AVA-Kinetics Localized Human Actions Video Dataset
• Focal Visual-Text Attention for Visual Question Answering
• What Objective Does Self-paced Learning Indeed Optimize?
• AC-VRNN: Attentive Conditional-VRNN for Multi-Future Trajectory Prediction
• Robust Neural Machine Translation with Doubly Adversarial Inputs
• Stochastic Prediction of Multi-Agent Interactions from Partial Observations
• Car that Knows Before You Do: Anticipating Maneuvers via Learning Temporal Driving Models
• Encouraging LSTMs to Anticipate Actions Very Early
• A Transferable Pedestrian Motion Prediction Model for Intersections with Different Geometries
• Actor-Context-Actor Relation Network for Spatio-Temporal Action Localization
• RSA: Randomized Simulation as Augmentation for Robust Human Action Recognition
• Technical Report of the Video Event Reconstruction and Analysis (VERA) System -- Shooter Localization, Models, Interface, and Beyond
• SCR-Graph: Spatial-Causal Relationships based Graph Reasoning Network for Human Action Prediction
• BGM: Building a Dynamic Guidance Map without Visual Images for Trajectory Prediction