PIC: Permutation Invariant Convolution for Recognizing Long-range Activities

Abstract

Neural operations as convolutions, self-attention, and vector aggregation are the go-to choices for recognizing short-range actions. However, they have three limitations in modeling long-range activities. This paper presents PIC, Permutation Invariant Convolution, a novel neural layer to model the temporal structure of long-range activities. It has three desirable properties. i. Unlike standard convolution, PIC is invariant to the temporal permutations of features within its receptive field, qualifying it to model the weak temporal structures. ii. Different from vector aggregation, PIC respects local connectivity, enabling it to learn long-range temporal abstractions using cascaded layers. iii. In contrast to self-attention, PIC uses shared weights, making it more capable of detecting the most discriminant visual evidence across long and noisy videos. We study the three properties of PIC and demonstrate its effectiveness in recognizing the long-range activities of Charades, Breakfast, and MultiThumos.

References in corpus (10)

• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• Convolutional Sequence to Sequence Learning
• The THUMOS Challenge on Action Recognition for Videos "in the Wild"
• Action Recognition using Visual Attention
• Learnable pooling with Context Gating for video classification
• Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
• Large Memory Layers with Product Keys
• VideoGraph: Recognizing Minutes-Long Human Activities in Videos

Cited by in corpus (4)

• Coarse to Fine Multi-Resolution Temporal Convolutional Network
• Technical Report: Temporal Aggregate Representations
• PGT: A Progressive Method for Training Models on Long Videos
• Temporal Action Segmentation with High-level Complex Activity Labels