MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning
arXiv:2005.05402
Abstract
Generating multi-sentence descriptions for videos is one of the most challenging captioning tasks due to its high requirements for not only visual relevance but also discourse-based coherence across the sentences in the paragraph. Towards this goal, we propose a new approach called Memory-Augmented Recurrent Transformer (MART), which uses a memory module to augment the transformer architecture. The memory module generates a highly summarized memory state from the video segments and the sentence history so as to help better prediction of the next sentence (w.r.t. coreference and repetition aspects), thus encouraging coherent paragraph generation. Extensive experiments, human evaluations, and qualitative analyses on two popular datasets ActivityNet Captions and YouCookII show that MART generates more coherent and less repetitive paragraph captions than baseline methods, while maintaining relevance to the input video events. All code is available open-source at: https://github.com/jayleicn/recurrent-transformer
ACL 2020 (12 pages)
References in corpus (7)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- On the difficulty of training Recurrent Neural Networks
- Coherent Multi-Sentence Video Description with Variable Level of Detail
- Towards Automatic Learning of Procedures from Web Instructional Videos
- UNITER: UNiversal Image-TExt Representation Learning
- CUHK & ETHZ & SIAT Submission to ActivityNet Challenge 2016
Cited by in corpus (5)
- CLIP-It! Language-Guided Video Summarization
- Sensor-Augmented Egocentric-Video Captioning with Dynamic Modal Attention
- CUPID: Adaptive Curation of Pre-training Data for Video-and-Language Representation Learning
- Learning Domain Adaptation with Model Calibration for Surgical Report Generation in Robotic Surgery
- Neuro-Symbolic Representations for Video Captioning: A Case for Leveraging Inductive Biases for Vision and Language