Publications (53)
TEMPER: Testing Emotional Perturbation in Quantitative Reasoning
Atahan Dokme, Benjamin Reichman, Larry Heck
Large language models are trained and evaluated on quantitative reasoning tasks written in clean, emotionally neutral language. However, real-world queries are often wrapped in fru…
Allo-AVA: A Large-Scale Multimodal Conversational AI Dataset for Allocentric Avatar Gesture Animation
Saif Punjwani, Larry Heck
The scarcity of high-quality, multimodal training data severely hinders the creation of lifelike avatar animations for conversational AI in virtual environments. Existing datasets…
RILOD: Near Real-Time Incremental Learning for Object Detection at the Edge
Dawei Li, Serafettin Tasci, Shalini Ghosh +3
Object detection models shipped with camera-equipped edge devices cannot cover the objects of interest for every user. Therefore, the incremental learning capability is a critical…
Domain Adaptation of Recurrent Neural Networks for Natural Language Understanding
Aaron Jaech, Larry Heck, Mari Ostendorf
The goal of this paper is to use multi-task learning to efficiently scale slot filling models for natural language understanding to handle multiple target tasks or domains. The key…
Class-incremental Learning via Deep Model Consolidation
Junting Zhang, Jie Zhang, Shalini Ghosh +5
Deep neural networks (DNNs) often suffer from "catastrophic forgetting" during incremental learning (IL) --- an abrupt degradation of performance on the original set of classes whe…
Latent-IM: Latent Interaction Management for Speech LLMs
Adar Avsian, Atahan Dokme, Tony Woo +1
The paper introduces Latent-IM, a framework that internally manages dialogue moves in speech‑based large language models by selecting and realizing conversational actions, achievin…
mForms : Multimodal Form-Filling with Question Answering
Larry Heck, Simon Heck, Anirudh Sundar
This paper presents a new approach to form-filling by reformulating the task as multimodal natural language Question Answering (QA). The reformulation is achieved by first translat…
A Unit Selection Methodology for Music Generation Using Deep Neural Networks
Mason Bretan, Gil Weinberg, Larry Heck
Several methods exist for a computer to generate music based on data including Markov chains, recurrent neural networks, recombinancy, and grammars. We explore the use of unit sele…
Building a Conversational Agent Overnight with Dialogue Self-Play
Pararth Shah, Dilek Hakkani-Tür, Gokhan Tür +4
We propose Machines Talking To Machines (M2M), a framework combining automation and crowdsourcing to rapidly bootstrap end-to-end dialogue agents for goal-oriented dialogues in arb…
Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems
Bing Liu, Gokhan Tur, Dilek Hakkani-Tur +2
In this work, we present a hybrid learning method for training task-oriented dialogue systems through online user interactions. Popular methods for learning task-oriented dialogues…
SNEAK: Evaluating Strategic Communication and Information Leakage in Large Language Models
Adar Avsian, Larry Heck
Large language models (LLMs) are increasingly deployed in multi-agent settings where communication must balance informativeness and secrecy. In such settings, an agent may need to…
LEGO: Language Model Building Blocks
Shrenik Bhansali, Alwin Jin, Tyler Lizzo +1
Large language models (LLMs) are essential in natural language processing (NLP) but are costly in data collection, pre-training, fine-tuning, and inference. Task-specific small lan…
QR-Erase: Efficient Subspace-Based Machine Unlearning with Layer Localization
Tyler Lizzo, Larry Heck
Machine unlearning seeks to remove targeted information from trained models without requiring costly retraining. Existing optimization-based methods often degrade unrelated capabil…
Weight-of-Thought Reasoning: Exploring Neural Network Weights for Enhanced LLM Reasoning
Saif Punjwani, Larry Heck
Large language models (LLMs) have demonstrated remarkable reasoning capabilities when prompted with strategies such as Chain-of-Thought (CoT). However, these approaches focus on to…
Reading with Intent -- Neutralizing Intent
Benjamin Reichman, Adar Avsian, Larry Heck
Queries to large language models (LLMs) can be divided into two parts: the instruction/question and the accompanying context. The context for retrieval-augmented generation (RAG) s…
SYNDICOM: Improving Conversational Commonsense with Error-Injection and Natural Language Feedback
Christopher Richardson, Anirudh Sundar, Larry Heck
Commonsense reasoning is a critical aspect of human communication. Despite recent advances in conversational AI driven by large language models, commonsense reasoning remains a cha…
Federated Control with Hierarchical Multi-Agent Deep Reinforcement Learning
Saurabh Kumar, Pararth Shah, Dilek Hakkani-Tur +1
We present a framework combining hierarchical and multi-agent deep reinforcement learning approaches to solve coordination problems among a multitude of agents using a semi-decentr…
Large Body Language Models
Saif Punjwani, Larry Heck
As virtual agents become increasingly prevalent in human-computer interaction, generating realistic and contextually appropriate gestures in real-time remains a significant challen…
gTBLS: Generating Tables from Text by Conditional Question Answering
Anirudh Sundar, Christopher Richardson, Larry Heck
Distilling large, unstructured text into a structured, condensed form such as tables is an open research problem. One of the primary challenges in automatically generating tables i…
Taking a HINT: Leveraging Explanations to Make Vision and Language Models More Grounded
Ramprasaath R. Selvaraju, Stefan Lee, Yilin Shen +5
Many vision and language models suffer from poor visual grounding - often falling back on easy-to-learn language priors rather than basing their decisions on visual concepts in the…
Outside Knowledge Conversational Video (OKCV) Dataset -- Dialoguing over Videos
Benjamin Reichman, Constantin Patsch, Jack Truxal +2
In outside knowledge visual question answering (OK-VQA), the model must identify relevant visual information within an image and incorporate external knowledge to accurately respon…
Emotions Where Art Thou: Understanding and Characterizing the Emotional Latent Space of Large Language Models
Benjamin Reichman, Adar Avsian, Larry Heck
This work investigates how large language models (LLMs) internally represent emotion by analyzing the geometry of their hidden-state space. The paper identifies a low-dimensional e…
Grounding Open-Domain Instructions to Automate Web Support Tasks
Nancy Xu, Sam Masling, Michael Du +4
Grounding natural language instructions on the web to perform previously unseen tasks enables accessibility and automation. We introduce a task and dataset to train AI agents from…
cTBLS: Augmenting Large Language Models with Conversational Tables
Anirudh S Sundar, Larry Heck
Optimizing accuracy and performance while eliminating hallucinations of open-domain conversational large language models (LLMs) is an open research challenge. A particularly promis…
end-to-end training of a large vocabulary end-to-end speech recognition system
Chanwoo Kim, Sungsoo Kim, Kwangyoun Kim +10
In this paper, we present an end-to-end training framework for building state-of-the-art end-to-end speech recognition systems. Our training system utilizes a cluster of Central Pr…
Draft, Verify, and Improve: Toward Training-Aware Speculative Decoding
Shrenik Bhansali, Larry Heck
Autoregressive (AR) decoding is a major latency bottleneck for large language models. Speculative decoding (SD) accelerates AR by letting a drafter propose multi-token blocks that…
UNLEARN Efficient Removal of Knowledge in Large Language Models
Tyler Lizzo, Larry Heck
Given the prevalence of large language models (LLMs) and the prohibitive cost of training these models from scratch, dynamically forgetting specific knowledge e.g., private or prop…
Leveraging Semantic Web Search and Browse Sessions for Multi-Turn Spoken Dialog Systems
Lu Wang, Larry Heck, Dilek Hakkani-Tur
Training statistical dialog models in spoken dialog systems (SDS) requires large amounts of annotated data. The lack of scalable methods for data mining and annotation poses a sign…
Unlearning in LLMs: Methods, Evaluation, and Open Challenges
Tyler Lizzo, Larry Heck
Large language models (LLMs) have achieved remarkable success across natural language processing tasks, yet their widespread deployment raises pressing concerns around privacy, cop…
Sequential Dialogue Context Modeling for Spoken Language Understanding
Ankur Bapna, Gokhan Tur, Dilek Hakkani-Tur +1
Spoken Language Understanding (SLU) is a key component of goal oriented dialogue systems that would parse user utterances into semantic frame representations. Traditionally SLU doe…
FLEx: Language Modeling with Few-shot Language Explanations
Adar Avsian, Christopher Richardson, Anirudh Sundar +1
Language models have become effective at a wide range of tasks, from math problem solving to open-domain question answering. However, they still make mistakes, and these mistakes a…
Towards Zero-Shot Frame Semantic Parsing for Domain Scaling
Ankur Bapna, Gokhan Tur, Dilek Hakkani-Tur +1
State-of-the-art slot filling models for goal-oriented human/machine conversational language understanding systems rely on deep learning methods. While multi-task training of such…
cPAPERS: A Dataset of Situated and Multimodal Interactive Conversations in Scientific Papers
Anirudh Sundar, Jin Xu, William Gay +2
An emerging area of research in situated and multimodal interactive conversations (SIMMC) includes interactions in scientific papers. Since scientific papers are primarily composed…
Leveraging Deep Neural Networks and Knowledge Graphs for Entity Disambiguation
Hongzhao Huang, Larry Heck, Heng Ji
Entity Disambiguation aims to link mentions of ambiguous entities to a knowledge base (e.g., Wikipedia). Modeling topical coherence is crucial for this task based on the assumption…
Emotion is Not Just a Label: Latent Emotional Factors in LLM Processing
Benjamin Reichman, Adar Avsian, Samuel Webster +1
Large language models are routinely deployed on text that varies widely in emotional tone, yet their reasoning behavior is typically evaluated without accounting for emotion as a s…
Commonsense Reasoning for Conversational AI: A Survey of the State of the Art
Christopher Richardson, Larry Heck
Large, transformer-based pretrained language models like BERT, GPT, and T5 have demonstrated a deep understanding of contextual semantics and language syntax. Their success has ena…
Scalable Multi-Domain Dialogue State Tracking
Abhinav Rastogi, Dilek Hakkani-Tur, Larry Heck
Dialogue state tracking (DST) is a key component of task-oriented dialogue systems. DST estimates the user's goal at each user turn given the interaction until then. State of the a…
Selective State-Space Adaptation and Retrieval for Language Model Reasoning
Atahan Dokme, Larry Heck
Low-rank adaptation introduces a static learned update applied identically to every input. The update provides task-level adaptation but does not explicitly represent token-level o…
iTBLS: A Dataset of Interactive Conversations Over Tabular Information
Anirudh Sundar, Christopher Richardson, Adar Avsian +1
This paper introduces Interactive Tables (iTBLS), a dataset of interactive conversations that focuses on natural-language manipulation of tabular information sourced from academic…
SensorQA: A Question Answering Benchmark for Daily-Life Monitoring
Benjamin Reichman, Xiaofan Yu, Lanxiang Hu +5
With the rapid growth in sensor data, effectively interpreting and interfacing with these data in a human-understandable way has become crucial. While existing research primarily f…
Learning and Evaluating Musical Features with Deep Autoencoders
Mason Bretan, Sageev Oore, Doug Eck +1
In this work we describe and evaluate methods to learn musical embeddings. Each embedding is a vector that represents four contiguous beats of music and is derived from a symbolic…
Contextual LSTM (CLSTM) models for Large scale NLP tasks
Shalini Ghosh, Oriol Vinyals, Brian Strope +3
Documents exhibit sequential structure at multiple levels of abstraction (e.g., sentences, paragraphs, sections). These abstractions constitute a natural hierarchy for representing…
Zero-Shot Learning for Semantic Utterance Classification
Yann N. Dauphin, Gokhan Tur, Dilek Hakkani-Tur +1
We propose a novel zero-shot learning method for semantic utterance classification (SUC). It learns a classifier for problems where none of the semantic categories …
Spoken Conversational Agents with Large Language Models
Chao-Han Huck Yang, Andreas Stolcke, Larry Heck
Spoken conversational agents are converging toward voice-native LLMs. This tutorial distills the path from cascaded ASR/NLU to end-to-end, retrieval-and vision-grounded systems. We…
End-to-End Optimization of Task-Oriented Dialogue Model with Deep Reinforcement Learning
Bing Liu, Gokhan Tur, Dilek Hakkani-Tur +2
In this paper, we present a neural network based task-oriented dialogue system that can be optimized end-to-end with deep reinforcement learning (RL). The system is able to track d…
Are Human Conversations Special? A Large Language Model Perspective
Toshish Jawale, Chaitanya Animesh, Sekhar Vallath +2
This study analyzes changes in the attention mechanisms of large language models (LLMs) when used to understand natural conversations between humans (human-human). We analyze three…
Generative Visual Dialogue System via Adaptive Reasoning and Weighted Likelihood Estimation
Heming Zhang, Shalini Ghosh, Larry Heck +4
The key challenge of generative Visual Dialogue (VD) systems is to respond to human queries with informative answers in natural and contiguous conversation flow. Traditional Maximu…
Multimodal Conversational AI: A Survey of Datasets and Approaches
Anirudh Sundar, Larry Heck
As humans, we experience the world with all our senses or modalities (sound, sight, touch, smell, and taste). We use these modalities, particularly sight and touch, to convey and i…
SensorChat: Answering Qualitative and Quantitative Questions during Long-Term Multimodal Sensor Interactions
Xiaofan Yu, Lanxiang Hu, Benjamin Reichman +5
Natural language interaction with sensing systems is crucial for addressing users' personal concerns and providing health-related insights into their daily lives. When a user asks…
Dense Passage Retrieval: Is it Retrieving?
Benjamin Reichman, Larry Heck
Dense passage retrieval (DPR) is the first step in the retrieval augmented generation (RAG) paradigm for improving the performance of large language models (LLM). DPR fine-tunes pr…
Reinforcement Learning via Auxiliary Task Distillation
Abhinav Narayan Harish, Larry Heck, Josiah P. Hanna +2
We present Reinforcement Learning via Auxiliary Task Distillation (AuxDistill), a new method that enables reinforcement learning (RL) to perform long-horizon robot control problems…
Evaluating Cross-Lingual Unlearning in Multilingual Language Models
Tyler Lizzo, Larry Heck
We present the first comprehensive evaluation of cross-lingual unlearning in multilingual LLMs. Using translated TOFU benchmarks in seven language/script variants, we test major un…
Emotional RAG LLMs: Reading Comprehension for the Open Internet
Benjamin Reichman, Adar Avsian, Kartik Talamadupula +2
Queries to large language models (LLMs) can be divided into two parts: the instruction/question and the accompanying context. The context for retrieval-augmented generation (RAG) s…