53 citations · 172 across the 14 of their papers we have counts for
16 papers
Style-transfer based Speech and Audio-visual Scene Understanding for Robot Action Sequence Acquisition from Videos
Chiori Hori, Puyuan Peng, David Harwath +7
To realize human-robot collaboration, robots need to execute actions for new tasks according to human instructions given finite prior knowledge. Human experts can share their knowl…
(2.5+1)D Spatio-Temporal Scene Graphs for Video Question Answering
Anoop Cherian, Chiori Hori, Tim K. Marks +1
Spatio-temporal scene-graph approaches to video-based reasoning tasks, such as video question-answering (QA), typically construct such graphs for every video frame. These approache…
Audio-Visual Scene-Aware Dialog and Reasoning using Audio-Visual Transformers with Joint Student-Teacher Learning
Ankit P. Shah, Shijie Geng, Peng Gao +5
In previous work, we have proposed the Audio-Visual Scene-Aware Dialog (AVSD) task, collected an AVSD dataset, developed AVSD technologies, and hosted an AVSD challenge track at bo…
Optimizing Latency for Online Video CaptioningUsing Audio-Visual Transformers
Chiori Hori, Takaaki Hori, Jonathan Le Roux
Video captioning is an essential technology to understand scenes and describe events in natural language. To apply it to real-time monitoring, a system needs not only to describe e…
Advanced Long-context End-to-end Speech Recognition Using Context-expanded Transformers
Takaaki Hori, Niko Moritz, Chiori Hori +1
This paper addresses end-to-end automatic speech recognition (ASR) for long audio recordings such as lecture and conversational speeches. Most end-to-end ASR models are designed to…
Multi-Pass Transformer for Machine Translation
Peng Gao, Chiori Hori, Shijie Geng +2
In contrast with previous approaches where information flows only towards deeper layers of a stack, we consider a multi-pass transformer (MPT) architecture in which earlier layers…
Dynamic Graph Representation Learning for Video Dialog via Multi-Modal Shuffled Transformers
Shijie Geng, Peng Gao, Moitreya Chatterjee +5
Given an input video, its associated audio, and a brief caption, the audio-visual scene aware dialog (AVSD) task requires an agent to indulge in a question-answer dialog with a hum…
Multi-Layer Content Interaction Through Quaternion Product For Visual Question Answering
Lei Shi, Shijie Geng, Kai Shuang +4
Multi-modality fusion technologies have greatly improved the performance of neural network-based Video Description/Caption, Visual Question Answering (VQA) and Audio Visual Scene-a…
Spatio-Temporal Ranked-Attention Networks for Video Captioning
Anoop Cherian, Jue Wang, Chiori Hori +1
Generating video descriptions automatically is a challenging task that involves a complex interplay between spatio-temporal visual features and language models. Given that videos c…
The Eighth Dialog System Technology Challenge
Seokhwan Kim, Michel Galley, Chulaka Gunasekara +18
This paper introduces the Eighth Dialog System Technology Challenge. In line with recent challenges, the eighth edition focuses on applying end-to-end dialog technologies in a prag…
Dialog System Technology Challenge 7
Koichiro Yoshino, Chiori Hori, Julien Perez +14
This paper introduces the Seventh Dialog System Technology Challenges (DSTC), which use shared datasets to explore the problem of building dialog systems. Recently, end-to-end dial…
Audio-Visual Scene-Aware Dialog
Huda Alamri, Vincent Cartillier, Abhishek Das +9
We introduce the task of scene-aware dialog. Our goal is to generate a complete and natural response to a question about a scene, given video and audio of the scene and the history…
End-to-End Audio Visual Scene-Aware Dialog using Multimodal Attention-Based Video Features
Chiori Hori, Huda Alamri, Jue Wang +10
Dialog systems need to understand dynamic visual scenes in order to have conversations with users about the objects and events around them. Scene-aware dialog systems for real-worl…
Audio Visual Scene-Aware Dialog (AVSD) Challenge at DSTC7
Huda Alamri, Vincent Cartillier, Raphael Gontijo Lopes +8
Scene-aware dialog systems will be able to have conversations with users about the objects and events around them. Progress on such systems can be made by integrating state-of-the-…
End-to-end Conversation Modeling Track in DSTC6
Chiori Hori, Takaaki Hori
End-to-end training of neural networks is a promising approach to automatic construction of dialog systems using a human-to-human dialog corpus. Recently, Vinyals et al. tested neu…
Attention-Based Multimodal Fusion for Video Description
Chiori Hori, Takaaki Hori, Teng-Yok Lee +3
Currently successful methods for video description are based on encoder-decoder sentence generation using recur-rent neural networks (RNNs). Recent work has shown the advantage of…