Attention-Based Multimodal Fusion for Video Description

Abstract

Currently successful methods for video description are based on encoder-decoder sentence generation using recur-rent neural networks (RNNs). Recent work has shown the advantage of integrating temporal and/or spatial attention mechanisms into these models, in which the decoder net-work predicts each word in the description by selectively giving more weight to encoded features from specific time frames (temporal attention) or to features from specific spatial regions (spatial attention). In this paper, we propose to expand the attention model to selectively attend not just to specific times or spatial regions, but to specific modalities of input such as image features, motion features, and audio features. Our new modality-dependent attention mechanism, which we call multimodal attention, provides a natural way to fuse multimodal information for video description. We evaluate our method on the Youtube2Text dataset, achieving results that are competitive with current state of the art. More importantly, we demonstrate that our model incorporating multimodal attention as well as temporal attention significantly outperforms the model that uses temporal attention alone.

Resubmitted to the rebuttal for CVPR 2017 for review, 8 pages, 4 figures

References in corpus (10)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Neural Machine Translation by Jointly Learning to Align and Translate
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• ADADELTA: An Adaptive Learning Rate Method
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Describing Videos by Exploiting Temporal Structure
• Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks
• Jointly Modeling Embedding and Translation to Bridge Video and Language

Cited by in corpus (7)

• Less Is More: Picking Informative Frames for Video Captioning
• Sensor-Augmented Egocentric-Video Captioning with Dynamic Modal Attention
• Move Forward and Tell: A Progressive Generator of Video Descriptions
• Integrating Scene Text and Visual Appearance for Fine-Grained Image Classification
• Sensor Transformation Attention Networks
• Multimodal Data Fusion based on the Global Workspace Theory
• Predicting and Attending to Damaging Collisions for Placing Everyday Objects in Photo-Realistic Simulations