Exploiting Temporal Coherence for Multi-modal Video Categorization
arXiv:2002.03844
Abstract
Multimodal ML models can process data in multiple modalities (e.g., video, images, audio, text) and are useful for video content analysis in a variety of problems (e.g., object detection, scene understanding). In this paper, we focus on the problem of video categorization by using a multimodal approach. We have developed a novel temporal coherence-based regularization approach, which applies to different types of models (e.g., RNN, NetVLAD, Transformer). We demonstrate through experiments how our proposed multimodal video categorization models with temporal coherence out-perform strong state-of-the-art baseline models.
References in corpus (5)
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- YouTube-8M: A Large-Scale Video Classification Benchmark
- Contextual LSTM (CLSTM) models for Large scale NLP tasks
- Multimodal Fusion with Deep Neural Networks for Audio-Video Emotion Recognition
- Generative Visual Dialogue System via Adaptive Reasoning and Weighted Likelihood Estimation