Bag of Visual Words and Fusion Methods for Action Recognition: Comprehensive Study and Good Practice

Abstract

Video based action recognition is one of the important and challenging problems in computer vision research. Bag of Visual Words model (BoVW) with local features has become the most popular method and obtained the state-of-the-art performance on several realistic datasets, such as the HMDB51, UCF50, and UCF101. BoVW is a general pipeline to construct a global representation from a set of local features, which is mainly composed of five steps: (i) feature extraction, (ii) feature pre-processing, (iii) codebook generation, (iv) feature encoding, and (v) pooling and normalization. Many efforts have been made in each step independently in different scenarios and their effect on action recognition is still unknown. Meanwhile, video data exhibits different views of visual pattern, such as static appearance and motion dynamics. Multiple descriptors are usually extracted to represent these different views. Many feature fusion methods have been developed in other areas and their influence on action recognition has never been investigated before. This paper aims to provide a comprehensive study of all steps in BoVW and different fusion methods, and uncover some good practice to produce a state-of-the-art action recognition system. Specifically, we explore two kinds of local features, ten kinds of encoding methods, eight kinds of pooling and normalization strategies, and three kinds of fusion methods. We conclude that every step is crucial for contributing to the final recognition rate. Furthermore, based on our comprehensive study, we propose a simple yet effective representation, called hybrid representation, by exploring the complementarity of different BoVW frameworks and local descriptors. Using this representation, we obtain the state-of-the-art on the three challenging datasets: HMDB51 (61.1%), UCF50 (92.3%), and UCF101 (87.9%).

References in corpus (1)

• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild

Cited by in corpus (31)

• Two-Stream Convolutional Networks for Action Recognition in Videos
• Action Recognition with Trajectory-Pooled Deep-Convolutional Descriptors
• Towards Good Practices for Very Deep Two-Stream ConvNets
• Convolutional Two-Stream Network Fusion for Video Action Recognition
• Learning Spatiotemporal Features with 3D Convolutional Networks
• Long-term Temporal Convolutions for Action Recognition
• CUHK & ETHZ & SIAT Submission to ActivityNet Challenge 2016
• A Comprehensive Study of Deep Video Action Recognition
• Evaluating Two-Stream CNN for Video Classification
• Human Action Recognition using Factorized Spatio-Temporal Convolutional Networks
• VideoLSTM Convolves, Attends and Flows for Action Recognition
• Action Recognition with Image Based CNN Features
• Real-time Action Recognition with Enhanced Motion Vector CNNs
• Temporal Pyramid Pooling Based Convolutional Neural Networks for Action Recognition
• A Discriminative CNN Video Representation for Event Detection
• Two Stream LSTM: A Deep Fusion Framework for Human Action Recognition
• Beyond Gaussian Pyramid: Multi-skip Feature Stacking for Action Recognition
• Crowd Counting via Weighted VLAD on Dense Attribute Feature Maps
• Handcrafted Local Features are Convolutional Neural Networks
• Sympathy for the Details: Dense Trajectories and Hybrid Classification Architectures for Action Recognition
• Pooling the Convolutional Layers in Deep ConvNets for Action Recognition
• The Best of Both Worlds: Combining Data-independent and Data-driven Approaches for Action Recognition
• Multi-Level Recurrent Residual Networks for Action Recognition
• Long-short Term Motion Feature for Action Classification and Retrieval
• Order-aware Convolutional Pooling for Video Based Action Recognition
• Hyper-Fisher Vectors for Action Recognition
• AVD: Adversarial Video Distillation
• Better Exploiting OS-CNNs for Better Event Recognition in Images
• Cross-modal Supervision for Learning Active Speaker Detection in Video
• Scene Flow to Action Map: A New Representation for RGB-D based Action Recognition with Convolutional Neural Networks
• Probing the Intra-Component Correlations within Fisher Vector for Material Classification