Multi-Task Zero-Shot Action Recognition with Prioritised Data Augmentation

Abstract

Zero-Shot Learning (ZSL) promises to scale visual recognition by bypassing the conventional model training requirement of annotated examples for every category. This is achieved by establishing a mapping connecting low-level features and a semantic description of the label space, referred as visual-semantic mapping, on auxiliary data. Reusing the learned mapping to project target videos into an embedding space thus allows novel-classes to be recognised by nearest neighbour inference. However, existing ZSL methods suffer from auxiliary-target domain shift intrinsically induced by assuming the same mapping for the disjoint auxiliary and target classes. This compromises the generalisation accuracy of ZSL recognition on the target data. In this work, we improve the ability of ZSL to generalise across this domain shift in both model- and data-centric ways by formulating a visual-semantic mapping with better generalisation properties and a dynamic data re-weighting method to prioritise auxiliary data that are relevant to the target classes. Specifically: (1) We introduce a multi-task visual-semantic mapping to improve generalisation by constraining the semantic mapping parameters to lie on a low-dimensional manifold, (2) We explore prioritised data augmentation by expanding the pool of auxiliary data with additional instances weighted by relevance to the target domain. The proposed new model is applied to the challenging zero-shot action recognition problem to demonstrate its advantages over existing ZSL models.

Published in ECCV 2016

References in corpus (9)

• Distributed Representations of Words and Phrases and their Compositionality
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Zero-Shot Learning Through Cross-Modal Transfer
• Evaluation of Output Embeddings for Fine-Grained Image Classification
• Transductive Multi-view Zero-Shot Learning
• Learning Task Grouping and Overlap in Multi-task Learning
• A Unified Perspective on Multi-Domain and Multi-Task Learning
• Semantic Embedding Space for Zero-Shot Action Recognition
• Reasoning about Linguistic Regularities in Word Embeddings using Matrix Manifolds

Cited by in corpus (17)

• A Comprehensive Study of Deep Video Action Recognition
• Zero-Shot Action Recognition in Videos: A Survey
• OmniArt: Multi-task Deep Learning for Artistic Data Analysis
• Action2Vec: A Crossmodal Embedding Approach to Action Learning
• TARN: Temporal Attentive Relation Network for Few-Shot and Zero-Shot Action Recognition
• Attributes' Importance for Zero-Shot Pose-Classification Based on Wearable Sensors
• All About Knowledge Graphs for Actions
• Towards Universal Representation for Unseen Action Recognition
• Tell me what you see: A zero-shot action recognition method based on natural language descriptions
• Learning to Learn from Noisy Web Videos
• GAN for Vision, KG for Relation: a Two-stage Deep Network for Zero-shot Action Recognition
• Action Recognition in Untrimmed Videos with Composite Self-Attention Two-Stream Framework
• Elaborative Rehearsal for Zero-shot Action Recognition
• Multi-Modal Zero-Shot Sign Language Recognition
• ZS-SLR: Zero-Shot Sign Language Recognition from RGB-D Videos
• Efficient Pipelines for Vision-Based Context Sensing
• Exploring Temporal Information for Improved Video Understanding