Publications (52)
Enabling Classifiers to Make Judgements Explicitly Aligned with Human Values
Yejin Bang, Tiezheng Yu, Andrea Madotto +3
Many NLP classification tasks, such as sexism/racism detection or toxicity detection, are based on human values. Yet, human values can vary under diverse cultural conditions. There…
Hierarchical Meta-Embeddings for Code-Switching Named Entity Recognition
Genta Indra Winata, Zhaojiang Lin, Jamin Shin +2
In countries that speak multiple main languages, mixing up different languages within a conversation is commonly called code-switching. Previous works addressing this challenge mai…
BiToD: A Bilingual Multi-Domain Dataset For Task-Oriented Dialogue Modeling
Zhaojiang Lin, Andrea Madotto, Genta Indra Winata +5
Task-oriented dialogue (ToD) benchmarks provide an important avenue to measure progress and develop better conversational agents. However, existing datasets for end-to-end ToD mode…
Joint Audio-Text Model for Expressive Speech-Driven 3D Facial Animation
Yingruo Fan, Zhaojiang Lin, Jun Saito +2
Speech-driven 3D facial animation with accurate lip synchronization has been widely studied. However, synthesizing realistic motions for the entire face during speech has rarely be…
Learning Fast Adaptation on Cross-Accented Speech Recognition
Genta Indra Winata, Samuel Cahyawijaya, Zihan Liu +4
Local dialects influence people to pronounce words of the same language differently from each other. The great variability and complex characteristics of accents creates a major ch…
Continual Learning in Task-Oriented Dialogue Systems
Andrea Madotto, Zhaojiang Lin, Zhenpeng Zhou +6
Continual learning in task-oriented dialogue systems can allow us to add new domains and functionalities through time without incurring the high cost of a whole system retraining.…
AnyMAL: An Efficient and Scalable Any-Modality Augmented Language Model
Seungwhan Moon, Andrea Madotto, Zhaojiang Lin +10
We present Any-Modality Augmented Language Model (AnyMAL), a unified model that reasons over diverse input modality signals (i.e. text, image, video, audio, IMU motion sensor), and…
Few-Shot Bot: Prompt-Based Learning for Dialogue Systems
Andrea Madotto, Zhaojiang Lin, Genta Indra Winata +1
Learning to converse using only a few examples is a great challenge in conversational AI. The current best conversational models, which are either good chit-chatters (e.g., Blender…
CAiRE: An Empathetic Neural Chatbot
Zhaojiang Lin, Peng Xu, Genta Indra Winata +4
In this paper, we present an end-to-end empathetic conversation agent CAiRE. Our system adapts TransferTransfo (Wolf et al., 2019) learning approach that fine-tunes a large-scale p…
Variational Transformers for Diverse Response Generation
Zhaojiang Lin, Genta Indra Winata, Peng Xu +2
Despite the great promise of Transformers in many sequence modeling tasks (e.g., machine translation), their deterministic nature hinders them from generalizing to high entropy tas…
SnapNTell: Enhancing Entity-Centric Visual Question Answering with Retrieval Augmented Multimodal LLM
Jielin Qiu, Andrea Madotto, Zhaojiang Lin +7
Vision-extended LLMs have made significant strides in Visual Question Answering (VQA). Despite these advancements, VLLMs still encounter substantial difficulties in handling querie…
Proactive Assistant Dialogue Generation from Streaming Egocentric Videos
Yichi Zhang, Xin Luna Dong, Zhaojiang Lin +5
Recent advances in conversational AI have been substantial, but developing real-time systems for perceptual task guidance remains challenging. These systems must provide interactiv…
Language Models are Few-shot Multilingual Learners
Genta Indra Winata, Andrea Madotto, Zhaojiang Lin +3
General-purpose language models have demonstrated impressive capabilities, performing on par with state-of-the-art approaches on a range of downstream natural language processing (…
WearVox: An Egocentric Multichannel Voice Assistant Benchmark for Wearables
Zhaojiang Lin, Yong Xu, Kai Sun +17
Wearable devices such as AI glasses are transforming voice assistants into always-available, hands-free collaborators that integrate seamlessly with daily life, but they also intro…
Stream RAG: Instant and Accurate Spoken Dialogue Systems with Streaming Tool Usage
Siddhant Arora, Haidar Khan, Kai Sun +14
End-to-end speech-in speech-out dialogue systems are emerging as a powerful alternative to traditional ASR-LLM-TTS pipelines, generating more natural, expressive responses with sig…
The Adapter-Bot: All-In-One Controllable Conversational Model
Andrea Madotto, Zhaojiang Lin, Yejin Bang +1
Considerable progress has been made towards conversational models that generate coherent and fluent responses by training large language models on large dialogue datasets. These mo…
On the Importance of Word Order Information in Cross-lingual Sequence Labeling
Zihan Liu, Genta Indra Winata, Samuel Cahyawijaya +3
Word order variances generally exist in different languages. In this paper, we hypothesize that cross-lingual models that fit into the word order of the source language might fail…
Plug-and-Play Conversational Models
Andrea Madotto, Etsuko Ishii, Zhaojiang Lin +2
There has been considerable progress made towards conversational models that generate coherent and fluent responses; however, this often involves training large language models on…
IMU2CLIP: Multimodal Contrastive Learning for IMU Motion Sensors from Egocentric Videos and Text
Seungwhan Moon, Andrea Madotto, Zhaojiang Lin +4
We present IMU2CLIP, a novel pre-training approach to align Inertial Measurement Unit (IMU) motion sensor recordings with video and text, by projecting them into the joint represen…
ConfRAG: Confidence-Guided Retrieval-Augmenting Generation
Yin Huang, Yifan Ethan Xu, Kai Sun +12
Can Large Language Models (LLMs) be trained to avoid hallucinating factual statements, and can Retrieval-Augmented Generation (RAG) be triggered only when necessary to reduce retri…
SCRIBES: Web-Scale Script-Based Semi-Structured Data Extraction with Reinforcement Learning
Shicheng Liu, Kai Sun, Lisheng Fu +8
Semi-structured content in HTML tables, lists, and infoboxes accounts for a substantial share of factual data on the web, yet the formatting complicates usage, and reliably extract…
Introducing Semantics into Speech Encoders
Derek Xu, Shuyan Dong, Changhan Wang +10
Recent studies find existing self-supervised speech encoders contain primarily acoustic rather than semantic information. As a result, pipelined supervised automatic speech recogni…
MoEL: Mixture of Empathetic Listeners
Zhaojiang Lin, Andrea Madotto, Jamin Shin +2
Previous research on empathetic dialogue systems has mostly focused on generating responses given certain emotions. However, being empathetic not only requires the ability of gener…
Learning Comment Generation by Leveraging User-Generated Data
Zhaojiang Lin, Genta Indra Winata, Pascale Fung
Existing models on open-domain comment generation are difficult to train, and they produce repetitive and uninteresting responses. The problem is due to multiple and contradictory…
Continual Dialogue State Tracking via Example-Guided Question Answering
Hyundong Cho, Andrea Madotto, Zhaojiang Lin +5
Dialogue systems are frequently updated to accommodate new services, but naively updating them by continually training with data for new services in diminishing performance on prev…
Zero-Shot Dialogue State Tracking via Cross-Task Transfer
Zhaojiang Lin, Bing Liu, Andrea Madotto +8
Zero-shot transfer learning for dialogue state tracking (DST) enables us to handle a variety of task-oriented dialogue domains without the expense of collecting in-domain data. In…
MinTL: Minimalist Transfer Learning for Task-Oriented Dialogue Systems
Zhaojiang Lin, Andrea Madotto, Genta Indra Winata +1
In this paper, we propose Minimalist Transfer Learning (MinTL) to simplify the system design process of task-oriented dialogue systems and alleviate the over-dependency on annotate…
Cross-lingual Spoken Language Understanding with Regularized Representation Alignment
Zihan Liu, Genta Indra Winata, Peng Xu +2
Despite the promising results of current cross-lingual models for spoken language understanding systems, they still suffer from imperfect cross-lingual representation alignments be…
Pixel-Grounded Retrieval for Knowledgeable Large Multimodal Models
Jeonghwan Kim, Renjie Tao, Sanat Sharma +8
Visual Question Answering (VQA) often requires coupling fine-grained perception with factual knowledge beyond the input image. Prior multimodal Retrieval-Augmented Generation (MM-R…
Leveraging Slot Descriptions for Zero-Shot Cross-Domain Dialogue State Tracking
Zhaojiang Lin, Bing Liu, Seungwhan Moon +7
Zero-shot cross-domain dialogue state tracking (DST) enables us to handle task-oriented dialogue in unseen domains without the expense of collecting in-domain data. In this paper,…
Knowledge Extraction on Semi-Structured Content: Does It Remain Relevant for Question Answering in the Era of LLMs?
Kai Sun, Yin Huang, Srishti Mehra +11
The advent of Large Language Models (LLMs) has significantly advanced web-based Question Answering (QA) systems over semi-structured content, raising questions about the continued…
Language Models as Few-Shot Learner for Task-Oriented Dialogue Systems
Andrea Madotto, Zihan Liu, Zhaojiang Lin +1
Task-oriented dialogue systems use four connected modules, namely, Natural Language Understanding (NLU), a Dialogue State Tracking (DST), Dialogue Policy (DP) and Natural Language…
CAiRE_HKUST at SemEval-2019 Task 3: Hierarchical Attention for Dialogue Emotion Classification
Genta Indra Winata, Andrea Madotto, Zhaojiang Lin +4
Detecting emotion from dialogue is a challenge that has not yet been extensively surveyed. One could consider the emotion of each dialogue turn to be independent, but in this paper…
FaceFormer: Speech-Driven 3D Facial Animation with Transformers
Yingruo Fan, Zhaojiang Lin, Jun Saito +2
Speech-driven 3D facial animation is challenging due to the complex geometry of human faces and the limited availability of 3D audio-visual data. Prior works typically focus on lea…
Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning
Zhaojiang Lin, Andrea Madotto, Pascale Fung
Fine-tuning pre-trained generative language models to down-stream language generation tasks has shown promising results. However, this comes with the cost of having a single, large…
Personalizing Dialogue Agents via Meta-Learning
Zhaojiang Lin, Andrea Madotto, Chien-Sheng Wu +1
Existing personalized dialogue models use human designed persona descriptions to improve dialogue consistency. Collecting such descriptions from existing dialogues is expensive and…
Learning Knowledge Bases with Parameters for Task-Oriented Dialogue Systems
Andrea Madotto, Samuel Cahyawijaya, Genta Indra Winata +4
Task-oriented dialogue systems are either modularized with separate dialogue state tracking (DST) and management steps or end-to-end trainable. In either case, the knowledge base (…
Getting To Know You: User Attribute Extraction from Dialogues
Chien-Sheng Wu, Andrea Madotto, Zhaojiang Lin +2
User attributes provide rich and useful information for user understanding, yet structured and easy-to-use attributes are often sparsely populated. In this paper, we leverage dialo…
EmoGraph: Capturing Emotion Correlations using Graph Networks
Peng Xu, Zihan Liu, Genta Indra Winata +2
Most emotion recognition methods tackle the emotion understanding task by considering individual emotion independently while ignoring their fuzziness nature and the interconnection…
AssoMem: Scalable Memory QA with Multi-Signal Associative Retrieval
Kai Zhang, Xinyuan Zhang, Ejaz Ahmed +11
Accurate recall from large scale memories remains a core challenge for memory augmented AI assistants performing question answering (QA), especially in similarity dense scenarios w…
Attention over Parameters for Dialogue Systems
Andrea Madotto, Zhaojiang Lin, Chien-Sheng Wu +2
Dialogue systems require a great deal of different but complementary expertise to assist, inform, and entertain humans. For example, different domains (e.g., restaurant reservation…
VisualLens: Personalization through Task-Agnostic Visual History
Wang Bill Zhu, Deqing Fu, Kai Sun +8
Existing recommendation systems either rely on user interaction logs, such as online shopping history for shopping recommendations, or focus on text signals. However, item-based hi…
Are Multilingual Models Effective in Code-Switching?
Genta Indra Winata, Samuel Cahyawijaya, Zihan Liu +3
Multilingual language models have shown decent performance in multilingual and cross-lingual natural language understanding tasks. However, the power of these multilingual models i…
Thinking in Directivity: Speech Large Language Model for Multi-Talker Directional Speech Recognition
Jiamin Xie, Ju Lin, Yiteng Huang +8
Recent studies have demonstrated that prompting large language models (LLM) with audio encodings enables effective speech recognition capabilities. However, the ability of Speech L…
CAiRE in DialDoc21: Data Augmentation for Information-Seeking Dialogue System
Etsuko Ishii, Yan Xu, Genta Indra Winata +5
Information-seeking dialogue systems, including knowledge identification and response generation, aim to respond to users with fluent, coherent, and informative responses based on…
Meta-Transfer Learning for Code-Switched Speech Recognition
Genta Indra Winata, Samuel Cahyawijaya, Zhaojiang Lin +3
An increasing number of people in the world today speak a mixed-language as a result of being multilingual. However, building a speech recognition system for code-switching remains…
Aligning Paralinguistic Understanding and Generation in Speech LLMs via Multi-Task Reinforcement Learning
Jingxiang Chen, Minseok Kim, Seong-Gyun Leem +13
Speech large language models (LLMs) observe paralinguistic cues such as prosody, emotion, and non-verbal sounds--crucial for intent understanding. However, leveraging these cues fa…
Large Language Models as Zero-shot Dialogue State Tracker through Function Calling
Zekun Li, Zhiyu Zoey Chen, Mike Ross +7
Large language models (LLMs) are increasingly prevalent in conversational systems due to their advanced understanding and generative capabilities in general contexts. However, thei…
Attention-Informed Mixed-Language Training for Zero-shot Cross-lingual Task-oriented Dialogue Systems
Zihan Liu, Genta Indra Winata, Zhaojiang Lin +2
Recently, data-driven task-oriented dialogue systems have achieved promising performance in English. However, developing dialogue systems that support low-resource languages remain…
Lightweight and Efficient End-to-End Speech Recognition Using Low-Rank Transformer
Genta Indra Winata, Samuel Cahyawijaya, Zhaojiang Lin +2
Highly performing deep neural networks come at the cost of computational complexity that limits their practicality for deployment on portable devices. We propose the low-rank trans…
Pisces: An Auto-regressive Foundation Model for Image Understanding and Generation
Zhiyang Xu, Jiuhai Chen, Zhaojiang Lin +10
Recent advances in large language models (LLMs) have enabled multimodal foundation models to tackle both image understanding and generation within a unified framework. Despite thes…
XPersona: Evaluating Multilingual Personalized Chatbot
Zhaojiang Lin, Zihan Liu, Genta Indra Winata +5
Personalized dialogue systems are an essential step toward better human-machine interaction. Existing personalized dialogue agents rely on properly designed conversational datasets…