Efficient Modelling Across Time of Human Actions and Interactions

Abstract

This thesis focuses on video understanding for human action and interaction recognition. We start by identifying the main challenges related to action recognition from videos and review how they have been addressed by current methods. Based on these challenges, and by focusing on the temporal aspect of actions, we argue that current fixed-sized spatio-temporal kernels in 3D convolutional neural networks (CNNs) can be improved to better deal with temporal variations in the input. Our contributions are based on the enlargement of the convolutional receptive fields through the introduction of spatio-temporal size-varying segments of videos, as well as the discovery of the local feature relevance over the entire video sequence. The resulting extracted features encapsulate information that includes the importance of local features across multiple temporal durations, as well as the entire video sequence. Subsequently, we study how we can better handle variations between classes of actions, by enhancing their feature differences over different layers of the architecture. The hierarchical extraction of features models variations of relatively similar classes the same as very dissimilar classes. Therefore, distinctions between similar classes are less likely to be modelled. The proposed approach regularises feature maps by amplifying features that correspond to the class of the video that is processed. We move away from class-agnostic networks and make early predictions based on feature amplification mechanism. The proposed approaches are evaluated on several benchmark action recognition datasets and show competitive results. In terms of performance, we compete with the state-of-the-art while being more efficient in terms of GFLOPs. Finally, we present a human-understandable approach aimed at providing visual explanations for features learned over spatio-temporal networks.

PhD thesis

References in corpus (39)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Two-Stream Convolutional Networks for Action Recognition in Videos
• Generative Adversarial Networks
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Neural Architecture Search with Reinforcement Learning
• WaveNet: A Generative Model for Raw Audio
• How transferable are features in deep neural networks?
• The Kinetics Human Action Video Dataset
• Striving for Simplicity: The All Convolutional Net
• The Effectiveness of Data Augmentation in Image Classification using Deep Learning
• Understanding Neural Networks Through Deep Visualization
• Understanding deep learning requires rethinking generalization
• YouTube-8M: A Large-Scale Video Classification Benchmark
• Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
• Layer Normalization
• Spatiotemporal Residual Networks for Video Action Recognition
• Visualizing Deep Neural Network Decisions: Prediction Difference Analysis
• Distilling a Neural Network Into a Soft Decision Tree
• Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks
• GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond
• Attentional Pooling for Action Recognition
• L2 Regularization versus Batch and Weight Normalization
• Would Mega-scale Datasets Further Enhance Spatiotemporal 3D CNNs?
• Adding noise to the input of a model trained with a regularized objective
• VideoLSTM Convolves, Attends and Flows for Action Recognition
• Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification
• TEA: Temporal Excitation and Aggregation for Action Recognition
• FineGym: A Hierarchical Video Dataset for Fine-grained Action Understanding
• Collaborative Spatio-temporal Feature Learning for Video Action Recognition
• TEINet: Towards an Efficient Architecture for Video Recognition
• AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition
• Grouped Spatial-Temporal Aggregation for Efficient Action Recognition
• Attention Clusters: Purely Attention Based Local Feature Integration for Video Classification
• MoViNets: Mobile Video Networks for Efficient Video Recognition
• ElderSim: A Synthetic Data Generation Platform for Human Action Recognition in Eldercare Applications
• Video Action Recognition Via Neural Architecture Searching