Lattice Long Short-Term Memory for Human Action Recognition

Abstract

Human actions captured in video sequences are three-dimensional signals characterizing visual appearance and motion dynamics. To learn action patterns, existing methods adopt Convolutional and/or Recurrent Neural Networks (CNNs and RNNs). CNN based methods are effective in learning spatial appearances, but are limited in modeling long-term motion dynamics. RNNs, especially Long Short-Term Memory (LSTM), are able to learn temporal motion dynamics. However, naively applying RNNs to video sequences in a convolutional manner implicitly assumes that motions in videos are stationary across different spatial locations. This assumption is valid for short-term motions but invalid when the duration of the motion is long. In this work, we propose Lattice-LSTM (L2STM), which extends LSTM by learning independent hidden state transitions of memory cells for individual spatial locations. This method effectively enhances the ability to model dynamics across time and addresses the non-stationary issue of long-term motion dynamics without significantly increasing the model complexity. Additionally, we introduce a novel multi-modal training procedure for training our network. Unlike traditional two-stream architectures which use RGB and optical flow information as input, our two-stream model leverages both modalities to jointly train both input gates and both forget gates in the network rather than treating the two streams as separate entities with no information about the other. We apply this end-to-end system to benchmark datasets (UCF-101 and HMDB-51) of human action recognition. Experiments show that on both datasets, our proposed method outperforms all existing ones that are based on LSTM and/or CNNs of similar model complexities.

ICCV2017

References in corpus (11)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Residual Learning for Image Recognition
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptors
• Fully Convolutional Networks for Semantic Segmentation
• Achieving Human Parity in Conversational Speech Recognition
• Convolutional Two-Stream Network Fusion for Video Action Recognition
• Action Recognition using Visual Attention
• Learning Spatiotemporal Features with 3D Convolutional Networks
• Human Action Recognition using Factorized Spatio-Temporal Convolutional Networks
• Real-time Action Recognition with Enhanced Motion Vector CNNs

Cited by in corpus (6)

• Early Action Prediction with Generative Adversarial Networks
• RGB-D-based Human Motion Recognition with Deep Learning: A Survey
• Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition
• Cross-Modal Message Passing for Two-stream Fusion
• Universal-to-Specific Framework for Complex Action Recognition
• Fine-grained Video Categorization with Redundancy Reduction Attention