Action2Vec: A Crossmodal Embedding Approach to Action Learning
arXiv:1901.00484
Abstract
We describe a novel cross-modal embedding space for actions, named Action2Vec, which combines linguistic cues from class labels with spatio-temporal features derived from video clips. Our approach uses a hierarchical recurrent network to capture the temporal structure of video features. We train our embedding using a joint loss that combines classification accuracy with similarity to Word2Vec semantics. We evaluate Action2Vec by performing zero shot action recognition and obtain state of the art results on three standard datasets. In addition, we present two novel analogy tests which quantify the extent to which our joint embedding captures distributional semantics. This is the first joint embedding space to combine verbs and action videos, and the first to be thoroughly evaluated with respect to its distributional semantics.
References in corpus (12)
- Adam: A Method for Stochastic Optimization
- Distributed Representations of Words and Phrases and their Compositionality
- Two-Stream Convolutional Networks for Action Recognition in Videos
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- The Kinetics Human Action Video Dataset
- Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models
- VQA: Visual Question Answering
- Multi-Task Zero-Shot Action Recognition with Prioritised Data Augmentation
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
- Word2VisualVec: Image and Video to Sentence Matching by Visual Feature Prediction
- Text2Shape: Generating Shapes from Natural Language by Learning Joint Embeddings
- Learning Attributes Equals Multi-Source Domain Generalization
Cited by in corpus (9)
- Towards Zero-shot Sign Language Recognition
- Zero-Shot Action Recognition in Videos: A Survey
- All About Knowledge Graphs for Actions
- Few-Shot Action Localization without Knowing Boundaries
- Multi-Modal Zero-Shot Sign Language Recognition
- Universal Prototype Transport for Zero-Shot Action Recognition and Localization
- ZS-SLR: Zero-Shot Sign Language Recognition from RGB-D Videos
- Rudder: A Cross Lingual Video and Text Retrieval Dataset
- Unifying Few- and Zero-Shot Egocentric Action Recognition