Multi-Task Zero-Shot Action Recognition with Prioritised Data Augmentation

Abstract

Zero-Shot Learning (ZSL) promises to scale visual recognition by bypassing the conventional model training requirement of annotated examples for every category. This is achieved by establishing a mapping connecting low-level features and a semantic description of the label space, referred as visual-semantic mapping, on auxiliary data. Reusing the learned mapping to project target videos into an embedding space thus allows novel-classes to be recognised by nearest neighbour inference. However, existing ZSL methods suffer from auxiliary-target domain shift intrinsically induced by assuming the same mapping for the disjoint auxiliary and target classes. This compromises the generalisation accuracy of ZSL recognition on the target data. In this work, we improve the ability of ZSL to generalise across this domain shift in both model- and data-centric ways by formulating a visual-semantic mapping with better generalisation properties and a dynamic data re-weighting method to prioritise auxiliary data that are relevant to the target classes. Specifically: (1) We introduce a multi-task visual-semantic mapping to improve generalisation by constraining the semantic mapping parameters to lie on a low-dimensional manifold, (2) We explore prioritised data augmentation by expanding the pool of auxiliary data with additional instances weighted by relevance to the target domain. The proposed new model is applied to the challenging zero-shot action recognition problem to demonstrate its advantages over existing ZSL models.

Published in ECCV 2016

References in corpus (5)

• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• Zero-Shot Learning Through Cross-Modal Transfer
• Transductive Multi-view Zero-Shot Learning
• Learning Task Grouping and Overlap in Multi-task Learning
• A Unified Perspective on Multi-Domain and Multi-Task Learning

Cited by in corpus (12)

• A Comprehensive Study of Deep Video Action Recognition
• OmniArt: Multi-task Deep Learning for Artistic Data Analysis
• Action2Vec: A Crossmodal Embedding Approach to Action Learning
• TARN: Temporal Attentive Relation Network for Few-Shot and Zero-Shot Action Recognition
• All About Knowledge Graphs for Actions
• Learning to Learn from Noisy Web Videos
• GAN for Vision, KG for Relation: a Two-stage Deep Network for Zero-shot Action Recognition
• Multi-Modal Zero-Shot Sign Language Recognition
• Elaborative Rehearsal for Zero-shot Action Recognition
• ZS-SLR: Zero-Shot Sign Language Recognition from RGB-D Videos
• Efficient Pipelines for Vision-Based Context Sensing
• Exploring Temporal Information for Improved Video Understanding