An Attempt towards Interpretable Audio-Visual Video Captioning
arXiv:1812.02872
Abstract
Automatically generating a natural language sentence to describe the content of an input video is a very challenging problem. It is an essential multimodal task in which auditory and visual contents are equally important. Although audio information has been exploited to improve video captioning in previous works, it is usually regarded as an additional feature fed into a black box fusion machine. How are the words in the generated sentences associated with the auditory and visual modalities? The problem is still not investigated. In this paper, we make the first attempt to design an interpretable audio-visual video captioning network to discover the association between words in sentences and audio-visual sequences. To achieve this, we propose a multimodal convolutional neural network-based audio-visual video captioning framework and introduce a modality-aware module for exploring modality selection during sentence generation. Besides, we collect new audio captioning and visual captioning datasets for further exploring the interactions between auditory and visual modalities for high-level video understanding. Extensive experiments demonstrate that the modality-aware module makes our model interpretable on modality selection during sentence generation. Even with the added interpretability, our video captioning network can still achieve comparable performance with recent state-of-the-art methods.
arXiv comment11 pages, 4 figures
References in corpus (21)
- Adam: A Method for Stochastic Optimization
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Deep Residual Learning for Image Recognition
- Microsoft COCO Captions: Data Collection and Evaluation Server
- SoundNet: Learning Sound Representations from Unlabeled Video
- Sequence to Sequence -- Video to Text
- Show and Tell: A Neural Image Caption Generator
- Frame- and Segment-Level Features and Candidate Pool Evaluation for Video Caption Generation
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
- Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition
- CIDEr: Consensus-based Image Description Evaluation
- Pervasive Attention: 2D Convolutional Neural Networks for Sequence-to-Sequence Prediction
- Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning
- Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning
- Reconstruction Network for Video Captioning
- Learning to Separate Object Sounds by Watching Unlabeled Video
- Weakly Supervised Dense Video Captioning
- Less Is More: Picking Informative Frames for Video Captioning
- Video Captioning via Hierarchical Reinforcement Learning
- Watch, Listen, and Describe: Globally and Locally Aligned Cross-Modal Attentions for Video Captioning
- Video Captioning with Guidance of Multimodal Latent Topics