Multi-Label Activity Recognition using Activity-specific Features and Activity Correlations

Abstract

Multi-label activity recognition is designed for recognizing multiple activities that are performed simultaneously or sequentially in each video. Most recent activity recognition networks focus on single-activities, that assume only one activity in each video. These networks extract shared features for all the activities, which are not designed for multi-label activities. We introduce an approach to multi-label activity recognition that extracts independent feature descriptors for each activity and learns activity correlations. This structure can be trained end-to-end and plugged into any existing network structures for video classification. Our method outperformed state-of-the-art approaches on four multi-label activity recognition datasets. To better understand the activity-specific features that the system generated, we visualized these activity-specific features in the Charades dataset.

References in corpus (10)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• A Short Note about Kinetics-600
• TSM: Temporal Shift Module for Efficient Video Understanding
• DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding
• Concurrent Activity Recognition with Multimodal CNN-LSTM Structure
• AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures
• Temporal Pyramid Network for Action Recognition