ZRIGF: An Innovative Multimodal Framework for Zero-Resource Image-Grounded Dialogue Generation

Abstract

Image-grounded dialogue systems benefit greatly from integrating visual information, resulting in high-quality response generation. However, current models struggle to effectively utilize such information in zero-resource scenarios, mainly due to the disparity between image and text modalities. To overcome this challenge, we propose an innovative multimodal framework, called ZRIGF, which assimilates image-grounded information for dialogue generation in zero-resource situations. ZRIGF implements a two-stage learning strategy, comprising contrastive pre-training and generative pre-training. Contrastive pre-training includes a text-image matching module that maps images and texts into a unified encoded vector space, along with a text-assisted masked image modeling module that preserves pre-training visual features and fosters further multimodal feature alignment. Generative pre-training employs a multimodal fusion module and an information transfer module to produce insightful responses based on harmonized multimodal representations. Comprehensive experiments conducted on both text-based and image-grounded dialogue datasets demonstrate ZRIGF's efficacy in generating contextually pertinent and informative responses. Furthermore, we adopt a fully zero-resource scenario in the image-grounded dialogue dataset to demonstrate our framework's robust generalization capabilities in novel domains. The code is available at https://github.com/zhangbo-nlp/ZRIGF.

arXiv commentACM Multimedia 2023 Accpeted, Repo: https://github.com/zhangbo-nlp/ZRIGF

References in corpus (9)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Sequence to Sequence Learning with Neural Networks
• Decoupled Weight Decay Regularization
• Unified Language Model Pre-training for Natural Language Understanding and Generation
• Conversational AI: The Science Behind the Alexa Prize
• Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generation
• Zero-Resource Knowledge-Grounded Dialogue Generation
• The Zero Resource Speech Challenge 2020: Discovering discrete subword and word units
• Multimodal Dialogue Response Generation

Cited by in corpus (1)

• Distilling Implicit Multimodal Knowledge into Large Language Models for Zero-Resource Dialogue Generation