Bridging Text and Video: A Universal Multimodal Transformer for Video-Audio Scene-Aware Dialog

Abstract

Audio-Visual Scene-Aware Dialog (AVSD) is a task to generate responses when chatting about a given video, which is organized as a track of the 8th Dialog System Technology Challenge (DSTC8). To solve the task, we propose a universal multimodal transformer and introduce the multi-task learning method to learn joint representations among different modalities as well as generate informative and fluent responses. Our method extends the natural language generation pre-trained model to multimodal dialogue generation task. Our system achieves the best performance in both objective and subjective evaluations in the challenge.

arXiv commentAccepted by AAAI2020 DSTC8 workshop. Ranked 1st in DSTC8-AVSD track

References in corpus (10)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• The Kinetics Human Action Video Dataset
• The Curious Case of Neural Text Degeneration
• Wizard of Wikipedia: Knowledge-Powered Conversational agents
• TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents
• UNITER: UNiversal Image-TExt Representation Learning
• VideoBERT: A Joint Model for Video and Language Representation Learning
• Incremental Transformer with Deliberation Decoder for Document Grounded Conversations
• A Dataset for Document Grounded Conversations

Cited by in corpus (8)

• Towards Expressive Communication with Internet Memes: A New Multimodal Conversation Dataset and Benchmark
• Learning Reasoning Paths over Semantic Graphs for Video-grounded Dialogues
• VisualMRC: Machine Reading Comprehension on Document Images
• VidTr: Video Transformer Without Convolutions
• Mounting Video Metadata on Transformer-based Language Model for Open-ended Video Question Answering
• DVD: A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialogue
• Transfer training from smaller language model
• VGNMN: Video-grounded Neural Module Network to Video-Grounded Language Tasks