Knowledge Integration Networks for Action Recognition
arXiv:2002.07471
Abstract
In this work, we propose Knowledge Integration Networks (referred as KINet) for video action recognition. KINet is capable of aggregating meaningful context features which are of great importance to identifying an action, such as human information and scene context. We design a three-branch architecture consisting of a main branch for action recognition, and two auxiliary branches for human parsing and scene recognition which allow the model to encode the knowledge of human and scene for action recognition. We explore two pre-trained models as teacher networks to distill the knowledge of human and scene for training the auxiliary tasks of KINet. Furthermore, we propose a two-level knowledge encoding mechanism which contains a Cross Branch Integration (CBI) module for encoding the auxiliary knowledge into medium-level convolutional features, and an Action Knowledge Graph (AKG) for effectively fusing high-level context information. This results in an end-to-end trainable framework where the three tasks can be trained collaboratively, allowing the model to compute strong context knowledge efficiently. The proposed KINet achieves the state-of-the-art performance on a large-scale action recognition benchmark Kinetics-400, with a top-1 accuracy of 77.8%. We further demonstrate that our KINet has strong capability by transferring the Kinetics-trained model to UCF-101, where it obtains 97.8% top-1 accuracy.
To appear in AAAI 2020
References in corpus (11)
- Distilling the Knowledge in a Neural Network
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- A simple neural network module for relational reasoning
- Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
- Knowledge Guided Disambiguation for Large-Scale Scene Classification with Multi-Resolution CNNs
- SlowFast Networks for Video Recognition
- Look into Person: Self-supervised Structure-sensitive Learning and A New Benchmark for Human Parsing
- ActivityNet Challenge 2017 Summary
- Geometric Pose Affordance: 3D Human Pose with Scene Constraints
- ECO: Efficient Convolutional Network for Online Video Understanding
- Actor-Action Semantic Segmentation with Region Masks