iPerceive: Applying Common-Sense Reasoning to Multi-Modal Dense Video Captioning and Video Question Answering
arXiv:2011.07735
Abstract
Most prior art in visual understanding relies solely on analyzing the "what" (e.g., event recognition) and "where" (e.g., event localization), which in some cases, fails to describe correct contextual relationships between events or leads to incorrect underlying visual attention. Part of what defines us as human and fundamentally different from machines is our instinct to seek causality behind any association, say an event Y that happened as a direct result of event X. To this end, we propose iPerceive, a framework capable of understanding the "why" between events in a video by building a common-sense knowledge base using contextual cues to infer causal relationships between objects in the video. We demonstrate the effectiveness of our technique using the dense video captioning (DVC) and video question answering (VideoQA) tasks. Furthermore, while most prior work in DVC and VideoQA relies solely on visual information, other modalities such as audio and speech are vital for a human observer's perception of an environment. We formulate DVC and VideoQA tasks as machine translation problems that utilize multiple modalities. By evaluating the performance of iPerceive DVC and iPerceive VideoQA on the ActivityNet Captions and TVQA datasets respectively, we show that our approach furthers the state-of-the-art. Code and samples are available at: iperceive.amanchadha.com.
13 pages, 6 figures, 4 tables, Project Page: https://iperceive.amanchadha.com
References in corpus (15)
- Sequence to Sequence Learning with Neural Networks
- HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
- Women also Snowboard: Overcoming Bias in Captioning Models
- Deep Reinforcement Learning for Visual Object Tracking in Videos
- DenseCap: Fully Convolutional Localization Networks for Dense Captioning
- CausalGAN: Learning Causal Implicit Generative Models with Adversarial Training
- Visual Causal Feature Learning
- TVQA: Localized, Compositional Video Question Answering
- Causal Induction from Visual Observations for Goal Directed Tasks
- Unbiased Scene Graph Generation from Biased Training
- Spatio-Temporal Graph for Video Captioning with Knowledge Distillation
- TVQA+: Spatio-Temporal Grounding for Video Question Answering
- More Grounded Image Captioning by Distilling Image-Text Matching Model
- Two Causal Principles for Improving Visual Dialog
- RUC+CMU: System Report for Dense Captioning Events in Videos