Interactive Fusion of Multi-level Features for Compositional Activity Recognition
arXiv:2012.05689
Abstract
To understand a complex action, multiple sources of information, including appearance, positional, and semantic features, need to be integrated. However, these features are difficult to be fused since they often differ significantly in modality and dimensionality. In this paper, we present a novel framework that accomplishes this goal by interactive fusion, namely, projecting features across different spaces and guiding it using an auxiliary prediction task. Specifically, we implement the framework in three steps, namely, positional-to-appearance feature extraction, semantic feature interaction, and semantic-to-positional prediction. We evaluate our approach on two action recognition datasets, Something-Something and Charades. Interactive fusion achieves consistent accuracy gain beyond off-the-shelf action recognition algorithms. In particular, on Something-Else, the compositional setting of Something-Something, interactive fusion reports a remarkable gain of 2.9% in terms of top-1 accuracy.
References in corpus (6)
- The Kinetics Human Action Video Dataset
- Unsupervised Learning of Video Representations using LSTMs
- Video (language) modeling: a baseline for generative models of natural videos
- Object Level Visual Reasoning in Videos
- Memory-augmented Dense Predictive Coding for Video Representation Learning
- Learning Long-term Visual Dynamics with Region Proposal Interaction Networks