HEAR: Hearing Enhanced Audio Response for Video-grounded Dialogue

Abstract

Video-grounded Dialogue (VGD) aims to answer questions regarding a given multi-modal input comprising video, audio, and dialogue history. Although there have been numerous efforts in developing VGD systems to improve the quality of their responses, existing systems are competent only to incorporate the information in the video and text and tend to struggle in extracting the necessary information from the audio when generating appropriate responses to the question. The VGD system seems to be deaf, and thus, we coin this symptom of current systems' ignoring audio data as a deaf response. To overcome the deaf response problem, Hearing Enhanced Audio Response (HEAR) framework is proposed to perform sensible listening by selectively attending to audio whenever the question requires it. The HEAR framework enhances the accuracy and audibility of VGD systems in a model-agnostic manner. HEAR is validated on VGD datasets (i.e., AVSD@DSTC7 and AVSD@DSTC8) and shows effectiveness with various VGD systems.

EMNLP 2023, 14 pages, 13 figures

References in corpus (11)

• Decoupled Weight Decay Regularization
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• The Kinetics Human Action Video Dataset
• Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks
• SCANet: Scene Complexity Aware Network for Weakly-Supervised Video Moment Retrieval
• Selective Query-guided Debiasing for Video Corpus Moment Retrieval
• Multi-step Joint-Modality Attention Network for Scene-Aware Dialogue System
• Entropy-Enhanced Multimodal Attention Model for Scene-Aware Dialogue Generation
• DSTC8-AVSD: Multimodal Semantic Transformer Network with Retrieval Style Word Generator
• Information-Theoretic Text Hallucination Reduction for Video-grounded Dialogue
• Learning Reasoning Paths over Semantic Graphs for Video-grounded Dialogues