Fusing Multi-Stream Deep Networks for Video Classification

Abstract

This paper studies deep network architectures to address the problem of video classification. A multi-stream framework is proposed to fully utilize the rich multimodal information in videos. Specifically, we first train three Convolutional Neural Networks to model spatial, short-term motion and audio clues respectively. Long Short Term Memory networks are then adopted to explore long-term temporal dynamics. With the outputs of the individual streams, we propose a simple and effective fusion method to generate the final predictions, where the optimal fusion weights are learned adaptively for each class, and the learning process is regularized by automatically estimated class relationships. Our contributions are two-fold. First, the proposed multi-stream framework is able to exploit multimodal features that are more comprehensive than those previously attempted. Second, we demonstrate that the adaptive fusion method using the class relationship as a regularizer outperforms traditional alternatives that estimate the weights in a "free" fashion. Our framework produces significantly better results than the state of the arts on two popular benchmarks, 92.2\% on UCF-101 (without using audio) and 84.9\% on Columbia Consumer Videos.

References in corpus (7)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Going Deeper with Convolutions
• Beyond Gaussian Pyramid: Multi-skip Feature Stacking for Action Recognition
• Using Web Co-occurrence Statistics for Improving Image Categorization

Cited by in corpus (9)

• DiscrimNet: Semi-Supervised Action Recognition from Videos using Generative Adversarial Networks
• Bidirectional Long-Short Term Memory for Video Description
• Zero-Shot Visual Recognition via Bidirectional Latent Embedding
• Sympathy for the Details: Dense Trajectories and Hybrid Classification Architectures for Action Recognition
• Event and Activity Recognition in Video Surveillance for Cyber-Physical Systems
• Motion Feature Network: Fixed Motion Filter for Action Recognition
• Object-Level Context Modeling For Scene Classification with Context-CNN
• MOD: A Deep Mixture Model with Online Knowledge Distillation for Large Scale Video Temporal Concept Localization
• A Multimodal Sentiment Dataset for Video Recommendation