Follow the Attention: Combining Partial Pose and Object Motion for Fine-Grained Action Detection

Abstract

Retailers have long been searching for ways to effectively understand their customers' behaviour in order to provide a smooth and pleasant shopping experience that attracts more customers everyday and maximises their revenue, consequently. Humans can flawlessly understand others' behaviour by combining different visual cues from activity to gestures and facial expressions. Empowering the computer vision systems to do so, however, is still an open problem due to its intrinsic challenges as well as extrinsic enforced difficulties like lack of publicly available data and unique environment conditions (wild). In this work, We emphasise on detecting the first and by far the most crucial cue in behaviour analysis; that is human activity detection in computer vision. To do so, we introduce a framework for integrating human pose and object motion to both temporally detect and classify the activities in a fine-grained manner (very short and similar activities). We incorporate partial human pose and interaction with the objects in a multi-stream neural network architecture to guide the spatiotemporal attention mechanism for more efficient activity recognition. To this end, in the absence of pose supervision, we propose to use the Generative Adversarial Network (GAN) to generate exact joint locations from noisy probability heat maps. Additionally, based on the intuition that complex actions demand more than one source of information to be identified even by humans, we integrate the second stream of object motion to our network as a prior knowledge that we quantitatively show improves the recognition results. We empirically show the capability of our approach by achieving state-of-the-art results on MERL shopping dataset. We further investigate the effectiveness of this approach on a new shopping dataset that we have collected to address existing shortcomings.

References in corpus (23)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Playing Atari with Deep Reinforcement Learning
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The THUMOS Challenge on Action Recognition for Videos "in the Wild"
• FlowNet: Learning Optical Flow with Convolutional Networks
• Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition
• Convolutional Two-Stream Network Fusion for Video Action Recognition
• Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
• Non-local Neural Networks
• PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume
• R-C3D: Region Convolutional 3D Network for Temporal Activity Detection
• GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose
• An Empirical Study of Spatial Attention Mechanisms in Deep Networks
• Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
• Temporal Context Network for Activity Localization in Videos
• TURN TAP: Temporal Unit Regression Network for Temporal Action Proposals
• CDC: Convolutional-De-Convolutional Networks for Precise Temporal Action Localization in Untrimmed Videos
• FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks
• An End-to-end 3D Convolutional Neural Network for Action Detection and Segmentation in Videos
• Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition
• Temporal Convolutional Networks for Action Segmentation and Detection
• Occlusion Aware Unsupervised Learning of Optical Flow
• Spatio-Temporal Action Detection with Cascade Proposal and Location Anticipation