OpenViDial: A Large-Scale, Open-Domain Dialogue Dataset with Visual Contexts
arXiv:2012.15015
Abstract
When humans converse, what a speaker will say next significantly depends on what he sees. Unfortunately, existing dialogue models generate dialogue utterances only based on preceding textual contexts, and visual contexts are rarely considered. This is due to a lack of a large-scale multi-module dialogue dataset with utterances paired with visual contexts. In this paper, we release {\bf OpenViDial}, a large-scale multi-module dialogue dataset. The dialogue turns and visual contexts are extracted from movies and TV series, where each dialogue turn is paired with the corresponding visual context in which it takes place. OpenViDial contains a total number of 1.1 million dialogue turns, and thus 1.1 million visual contexts stored in images. Based on this dataset, we propose a family of encoder-decoder models leveraging both textual and visual contexts, from coarse-grained image features extracted from CNNs to fine-grained object features extracted from Faster R-CNNs. We observe that visual information significantly improves dialogue generation qualities, verifying the necessity of integrating multi-modal features for dialogue learning. Our work marks an important step towards large-scale multi-modal dialogue learning.
arXiv commentDataset, visual features and code are found at https://github.com/ShannonAI/OpenViDial
References in corpus (44)
- Adam: A Method for Stochastic Optimization
- Sequence to Sequence Learning with Neural Networks
- A Neural Conversational Model
- VisualBERT: A Simple and Performant Baseline for Vision and Language
- Hierarchical Question-Image Co-Attention for Visual Question Answering
- VQA: Visual Question Answering
- Deep Reinforcement Learning for Dialogue Generation
- Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs
- Topic Aware Neural Response Generation
- Towards a Human-like Open-Domain Chatbot
- LXMERT: Learning Cross-Modality Encoder Representations from Transformers
- Adversarial Learning for Neural Dialogue Generation
- Personalizing Dialogue Agents: I have a dog, do you have pets too?
- UNITER: UNiversal Image-TExt Representation Learning
- A Persona-Based Neural Conversation Model
- Data Noising as Smoothing in Neural Network Language Models
- Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoders
- Emotional Chatting Machine: Emotional Conversation Generation with Internal and External Memory
- DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation
- Visual Reference Resolution using Attention Memory for Visual Dialog
- Personalized Dialogue Generation with Diversified Traits
- A Neural Network Approach to Context-Sensitive Generation of Conversational Responses
- Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems
- Adversarial Evaluation of Dialogue Models
- Multi-Task Learning for Speaker-Role Adaptation in Neural Conversation Models
- DP-GAN: Diversity-Promoting Generative Adversarial Network for Generating Informative and Diversified Text
- CLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialog
- ConvLab: Multi-Domain End-to-End Dialog System Platform
- VD-BERT: A Unified Vision and Dialog Transformer with BERT
- Learning to Detect Relevant Contexts and Knowledge for Response Selection in Retrieval-based Dialogue Systems
- Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog
- MUREL: Multimodal Relational Reasoning for Visual Question Answering
- Rethinking Action Spaces for Reinforcement Learning in End-to-end Dialog Agents with Latent Variable Models
- Generating High-Quality and Informative Conversation Responses with Sequence-to-Sequence Models
- Entropy-Enhanced Multimodal Attention Model for Scene-Aware Dialogue Generation
- FlipDial: A Generative Model for Two-Way Visual Dialogue
- Steering Output Style and Topic in Neural Response Generation
- Dual Attention Networks for Visual Reference Resolution in Visual Dialog
- Making History Matter: History-Advantage Sequence Training for Visual Dialog
- Non-Autoregressive Neural Dialogue Generation
- Context, Attention and Audio Feature Explorations for Audio Visual Scene-Aware Dialog
- DualVD: An Adaptive Dual Encoding Model for Deep Visual Understanding in Visual Dialogue
- Polite Dialogue Generation Without Parallel Data
- Teaching Machines to Converse