Efficient Modelling Across Time of Human Actions and Interactions

Abstract

This thesis focuses on video understanding for human action and interaction recognition. We start by identifying the main challenges related to action recognition from videos and review how they have been addressed by current methods. Based on these challenges, and by focusing on the temporal aspect of actions, we argue that current fixed-sized spatio-temporal kernels in 3D convolutional neural networks (CNNs) can be improved to better deal with temporal variations in the input. Our contributions are based on the enlargement of the convolutional receptive fields through the introduction of spatio-temporal size-varying segments of videos, as well as the discovery of the local feature relevance over the entire video sequence. The resulting extracted features encapsulate information that includes the importance of local features across multiple temporal durations, as well as the entire video sequence. Subsequently, we study how we can better handle variations between classes of actions, by enhancing their feature differences over different layers of the architecture. The hierarchical extraction of features models variations of relatively similar classes the same as very dissimilar classes. Therefore, distinctions between similar classes are less likely to be modelled. The proposed approach regularises feature maps by amplifying features that correspond to the class of the video that is processed. We move away from class-agnostic networks and make early predictions based on feature amplification mechanism. The proposed approaches are evaluated on several benchmark action recognition datasets and show competitive results. In terms of performance, we compete with the state-of-the-art while being more efficient in terms of GFLOPs. Finally, we present a human-understandable approach aimed at providing visual explanations for features learned over spatio-temporal networks.

arXiv commentPhD thesis

References in corpus (90)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Auto-Encoding Variational Bayes
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Two-Stream Convolutional Networks for Action Recognition in Videos
• Generative Adversarial Networks
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Neural Architecture Search with Reinforcement Learning
• WaveNet: A Generative Model for Raw Audio
• How transferable are features in deep neural networks?
• Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Axiomatic Attribution for Deep Networks
• The Kinetics Human Action Video Dataset
• Striving for Simplicity: The All Convolutional Net
• The Effectiveness of Data Augmentation in Image Classification using Deep Learning
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Learning Important Features Through Propagating Activation Differences
• Squeeze-and-Excitation Networks
• Wide Residual Networks
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Understanding Neural Networks Through Deep Visualization
• Explaining NonLinear Classification Decisions with Deep Taylor Decomposition
• OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks
• Understanding deep learning requires rethinking generalization
• YouTube-8M: A Large-Scale Video Classification Benchmark
• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Visualizing and Understanding Recurrent Networks
• The THUMOS Challenge on Action Recognition for Videos "in the Wild"
• Efficient Neural Architecture Search via Parameter Sharing
• Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
• Unstructured Human Activity Detection from RGBD Images
• Gather-Excite: Exploiting Feature Context in Convolutional Neural Networks
• Layer Normalization
• Spatiotemporal Residual Networks for Video Action Recognition
• Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
• A Short Note on the Kinetics-700-2020 Human Action Dataset
• A Short Note about Kinetics-600
• Scaling Egocentric Vision: The EPIC-KITCHENS Dataset
• Distilling a Neural Network Into a Soft Decision Tree
• Dropout Training as Adaptive Regularization
• Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks
• Synthesizing the preferred inputs for neurons in neural networks via deep generator networks
• How Does Batch Normalization Help Optimization?
• A Closer Look at Spatiotemporal Convolutions for Action Recognition
• GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond
• Attentional Pooling for Action Recognition
• L2 Regularization versus Batch and Weight Normalization
• RISE: Randomized Input Sampling for Explanation of Black-box Models
• TSM: Temporal Shift Module for Efficient Video Understanding
• Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networks
• Drop an Octave: Reducing Spatial Redundancy in Convolutional Neural Networks with Octave Convolution
• Long-term Temporal Convolutions for Action Recognition
• Attention to Scale: Scale-aware Semantic Image Segmentation
• Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?
• More Is Less: Learning Efficient Video Representations by Big-Little Network and Depthwise Temporal Aggregation
• Score-CAM: Score-Weighted Visual Explanations for Convolutional Neural Networks
• Video Classification with Channel-Separated Convolutional Networks
• Would Mega-scale Datasets Further Enhance Spatiotemporal 3D CNNs?
• Analyzing Human-Human Interactions: A Survey
• Human Action Localization with Sparse Spatial Supervision
• Adding noise to the input of a model trained with a regularized objective
• VideoLSTM Convolves, Attends and Flows for Action Recognition
• STM: SpatioTemporal and Motion Encoding for Action Recognition
• Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification
• SS-CAM: Smoothed Score-CAM for Sharper Visual Feature Localization
• TEA: Temporal Excitation and Aggregation for Action Recognition
• Collaborative Spatio-temporal Feature Learning for Video Action Recognition
• IS-CAM: Integrated Score-CAM for axiomatic-based explanations
• FineGym: A Hierarchical Video Dataset for Fine-grained Action Understanding
• Learn to cycle: Time-consistent feature discovery for action recognition
• TEINet: Towards an Efficient Architecture for Video Recognition
• AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures
• Spatio-Temporal FAST 3D Convolutions for Human Action Recognition
• AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition
• Grouped Spatial-Temporal Aggregation for Efficient Action Recognition
• AViD Dataset: Anonymized Videos from Diverse Countries
• Attention Clusters: Purely Attention Based Local Feature Integration for Video Classification
• MoViNets: Mobile Video Networks for Efficient Video Recognition
• On the Units of GANs (Extended Abstract)
• ElderSim: A Synthetic Data Generation Platform for Human Action Recognition in Eldercare Applications
• HACS: Human Action Clips and Segments Dataset for Recognition and Temporal Localization
• Discriminating Spatial and Temporal Relevance in Deep Taylor Decompositions for Explainable Activity Recognition
• Explaining Motion Relevance for Activity Recognition in Video Deep Learning Models
• Video Action Recognition Via Neural Architecture Searching
• Towards Visually Explaining Video Understanding Networks with Perturbation
• Learning Class Regularized Features for Action Recognition
• Saliency guided deep network for weakly-supervised image segmentation