Context, Attention and Audio Feature Explorations for Audio Visual Scene-Aware Dialog

Abstract

With the recent advancements in AI, Intelligent Virtual Assistants (IVA) have become a ubiquitous part of every home. Going forward, we are witnessing a confluence of vision, speech and dialog system technologies that are enabling the IVAs to learn audio-visual groundings of utterances and have conversations with users about the objects, activities and events surrounding them. As a part of the 7th Dialog System Technology Challenges (DSTC7), for Audio Visual Scene-Aware Dialog (AVSD) track, We explore `topics' of the dialog as an important contextual feature into the architecture along with explorations around multimodal Attention. We also incorporate an end-to-end audio classification ConvNet, AclNet, into our models. We present detailed analysis of the experiments and show that some of our model variations outperform the baseline system presented for this task.

7 pages, 2 figures, DSTC7 workshop at AAAI 2019

References in corpus (7)

• Neural Machine Translation by Jointly Learning to Align and Translate
• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• VQA: Visual Question Answering
• Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
• Sequence to Sequence -- Video to Text
• Describing Videos by Exploiting Temporal Structure
• AclNet: efficient end-to-end audio classification CNN

Cited by in corpus (5)

• OpenViDial: A Large-Scale, Open-Domain Dialogue Dataset with Visual Contexts
• Leveraging Topics and Audio Features with Multimodal Attention for Audio Visual Scene-Aware Dialog
• TMT: A Transformer-based Modal Translator for Improving Multimodal Sequence Representations in Audio Visual Scene-aware Dialog
• Exploring Context, Attention and Audio Features for Audio Visual Scene-Aware Dialog
• Learning Question-Guided Video Representation for Multi-Turn Video Question Answering