Bridging Text and Video: A Universal Multimodal Transformer for Video-Audio Scene-Aware Dialog

Abstract

Audio-Visual Scene-Aware Dialog (AVSD) is a task to generate responses when chatting about a given video, which is organized as a track of the 8th Dialog System Technology Challenge (DSTC8). To solve the task, we propose a universal multimodal transformer and introduce the multi-task learning method to learn joint representations among different modalities as well as generate informative and fluent responses. Our method extends the natural language generation pre-trained model to multimodal dialogue generation task. Our system achieves the best performance in both objective and subjective evaluations in the challenge.

Accepted by AAAI2020 DSTC8 workshop. Ranked 1st in DSTC8-AVSD track

References in corpus (3)

• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• The Kinetics Human Action Video Dataset
• TransferTransfo: A Transfer Learning Approach for Neural Network Based Conversational Agents

Cited by in corpus (3)

• Towards Expressive Communication with Internet Memes: A New Multimodal Conversation Dataset and Benchmark
• Mounting Video Metadata on Transformer-based Language Model for Open-ended Video Question Answering
• Transfer training from smaller language model