Bag of Visual Words and Fusion Methods for Action Recognition: Comprehensive Study and Good Practice

Abstract

Video based action recognition is one of the important and challenging problems in computer vision research. Bag of Visual Words model (BoVW) with local features has become the most popular method and obtained the state-of-the-art performance on several realistic datasets, such as the HMDB51, UCF50, and UCF101. BoVW is a general pipeline to construct a global representation from a set of local features, which is mainly composed of five steps: (i) feature extraction, (ii) feature pre-processing, (iii) codebook generation, (iv) feature encoding, and (v) pooling and normalization. Many efforts have been made in each step independently in different scenarios and their effect on action recognition is still unknown. Meanwhile, video data exhibits different views of visual pattern, such as static appearance and motion dynamics. Multiple descriptors are usually extracted to represent these different views. Many feature fusion methods have been developed in other areas and their influence on action recognition has never been investigated before. This paper aims to provide a comprehensive study of all steps in BoVW and different fusion methods, and uncover some good practice to produce a state-of-the-art action recognition system. Specifically, we explore two kinds of local features, ten kinds of encoding methods, eight kinds of pooling and normalization strategies, and three kinds of fusion methods. We conclude that every step is crucial for contributing to the final recognition rate. Furthermore, based on our comprehensive study, we propose a simple yet effective representation, called hybrid representation, by exploring the complementarity of different BoVW frameworks and local descriptors. Using this representation, we obtain the state-of-the-art on the three challenging datasets: HMDB51 (61.1%), UCF50 (92.3%), and UCF101 (87.9%).

References in corpus (1)

• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild

Cited by in corpus (15)

• Two-Stream Convolutional Networks for Action Recognition in Videos
• Towards Good Practices for Very Deep Two-Stream ConvNets
• CUHK & ETHZ & SIAT Submission to ActivityNet Challenge 2016
• A Comprehensive Study of Deep Video Action Recognition
• Evaluating Two-Stream CNN for Video Classification
• VideoLSTM Convolves, Attends and Flows for Action Recognition
• Temporal Pyramid Pooling Based Convolutional Neural Networks for Action Recognition
• A Discriminative CNN Video Representation for Event Detection
• Beyond Gaussian Pyramid: Multi-skip Feature Stacking for Action Recognition
• Sympathy for the Details: Dense Trajectories and Hybrid Classification Architectures for Action Recognition
• Long-short Term Motion Feature for Action Classification and Retrieval
• Multi-Level Recurrent Residual Networks for Action Recognition
• The Best of Both Worlds: Combining Data-independent and Data-driven Approaches for Action Recognition
• AVD: Adversarial Video Distillation
• Scene Flow to Action Map: A New Representation for RGB-D based Action Recognition with Convolutional Neural Networks