83 citations · 250 across the 21 of their papers we have counts for
23 papers
Social network modeling and applications, a tutorial
Lisette Espín-Noboa, Tiago Peixoto, Fariba Karimi
Social networks have been widely studied over the last century from multiple disciplines to understand societal issues such as inequality in employment rates, managerial performanc…
Embodied Executable Policy Learning with Language-based Scene Summarization
Jielin Qiu, Mengdi Xu, William Han +2
Large Language models (LLMs) have shown remarkable success in assisting robot learning tasks, i.e., complex household planning. However, the performance of pretrained LLMs heavily…
Normalized Contrastive Learning for Text-Video Retrieval
Yookoon Park, Mahmoud Azab, Bo Xiong +4
Cross-modal contrastive learning has led the recent advances in multimodal retrieval with its simplicity and effectiveness. In this work, however, we reveal that cross-modal contra…
Navigating Connected Memories with a Task-oriented Dialog System
Seungwhan Moon, Satwik Kottur, Alborz Geramifard +1
Recent years have seen an increasing trend in the volume of personal media captured by users, thanks to the advent of smartphones and smart glasses, resulting in large media collec…
Tell Your Story: Task-Oriented Dialogs for Interactive Content Creation
Satwik Kottur, Seungwhan Moon, Aram H. Markosyan +3
People capture photos and videos to relive and share memories of personal significance. Recently, media montages (stories) have become a popular mode of sharing these memories due…
IMU2CLIP: Multimodal Contrastive Learning for IMU Motion Sensors from Egocentric Videos and Text
Seungwhan Moon, Andrea Madotto, Zhaojiang Lin +4
We present IMU2CLIP, a novel pre-training approach to align Inertial Measurement Unit (IMU) motion sensor recordings with video and text, by projecting them into the joint represen…
Fighting FIRe with FIRE: Assessing the Validity of Text-to-Video Retrieval Benchmarks
Pedro Rodriguez, Mahmoud Azab, Becka Silvert +4
Searching troves of videos with textual descriptions is a core multimodal retrieval task. Owing to the lack of a purpose-built dataset for text-to-video retrieval, video captioning…
KETOD: Knowledge-Enriched Task-Oriented Dialogue
Zhiyu Chen, Bing Liu, Seungwhan Moon +3
Existing studies in dialogue system research mostly treat task-oriented dialogue and chit-chat as separate domains. Towards building a human-like assistant that can converse natura…
Zero-Shot Dialogue State Tracking via Cross-Task Transfer
Zhaojiang Lin, Bing Liu, Andrea Madotto +8
Zero-shot transfer learning for dialogue state tracking (DST) enables us to handle a variety of task-oriented dialogue domains without the expense of collecting in-domain data. In…
Leveraging Slot Descriptions for Zero-Shot Cross-Domain Dialogue State Tracking
Zhaojiang Lin, Bing Liu, Seungwhan Moon +7
Zero-shot cross-domain dialogue state tracking (DST) enables us to handle task-oriented dialogue in unseen domains without the expense of collecting in-domain data. In this paper,…
SIMMC 2.0: A Task-oriented Dialog Dataset for Immersive Multimodal Conversations
Satwik Kottur, Seungwhan Moon, Alborz Geramifard +1
Next generation task-oriented dialog systems need to understand conversational contexts with their perceived surroundings, to effectively help users in the real-world multimodal en…
DVD: A Diagnostic Dataset for Multi-step Reasoning in Video Grounded Dialogue
Hung Le, Chinnadhurai Sankar, Seungwhan Moon +3
A video-grounded dialogue system is required to understand both dialogue, which contains semantic dependencies from turn to turn, and video, which contains visual cues of spatial a…
Continual Learning in Task-Oriented Dialogue Systems
Andrea Madotto, Zhaojiang Lin, Zhenpeng Zhou +6
Continual learning in task-oriented dialogue systems can allow us to add new domains and functionalities through time without incurring the high cost of a whole system retraining.…
Overview of the Ninth Dialog System Technology Challenge: DSTC9
Chulaka Gunasekara, Seokhwan Kim, Luis Fernando D'Haro +36
This paper introduces the Ninth Dialog System Technology Challenge (DSTC-9). This edition of the DSTC focuses on applying end-to-end dialog technologies for four distinct tasks in…
NUANCED: Natural Utterance Annotation for Nuanced Conversation with Estimated Distributions
Zhiyu Chen, Honglei Liu, Hu Xu +3
Existing conversational systems are mostly agent-centric, which assumes the user utterances would closely follow the system ontology (for NLU or dialogue state tracking). However,…
Adding Chit-Chat to Enhance Task-Oriented Dialogues
Kai Sun, Seungwhan Moon, Paul Crook +7
Existing dialogue corpora and models are typically designed under two disjoint motives: while task-oriented systems focus on achieving functional goals (e.g., booking hotels), open…
Situated and Interactive Multimodal Conversations
Seungwhan Moon, Satwik Kottur, Paul A. Crook +9
Next generation virtual assistants are envisioned to handle multimodal inputs (e.g., vision, memories of previous interactions, in addition to the user's utterances), and perform m…
User Memory Reasoning for Conversational Recommendation
Hu Xu, Seungwhan Moon, Honglei Liu +3
We study a conversational recommendation model which dynamically manages users' past (offline) preferences and current (online) requests through a structured and cumulative user me…
Information Seeking in the Spirit of Learning: a Dataset for Conversational Curiosity
Pedro Rodriguez, Paul Crook, Seungwhan Moon +1
Open-ended human learning and information-seeking are increasingly mediated by digital assistants. However, such systems often ignore the user's pre-existing knowledge. Assuming a…
Active Federated Learning
Jack Goetz, Kshitiz Malik, Duc Bui +3
Federated Learning allows for population level models to be trained without centralizing client data by transmitting the global model to clients, calculating gradients locally, the…
Federated User Representation Learning
Duc Bui, Kshitiz Malik, Jack Goetz +4
Collaborative personalization, such as through learned user representations (embeddings), can improve the prediction accuracy of neural-network-based models significantly. We propo…
Multimodal Named Entity Recognition for Short Social Media Posts
Seungwhan Moon, Leonardo Neves, Vitor Carvalho
We introduce a new task called Multimodal Named Entity Recognition (MNER) for noisy user-generated data such as tweets or Snapchat captions, which comprise short text with accompan…
Multimodal Transfer Deep Learning with Applications in Audio-Visual Recognition
Seungwhan Moon, Suyoun Kim, Haohan Wang
We propose a transfer deep learning (TDL) framework that can transfer the knowledge obtained from a single-modal neural network to a network with a different modality. Specifically…