14 citations · 32 across the 28 of their papers we have counts for
29 papers
Which Metrics Save the Most Human Annotation? Prediction-Powered Evaluation and Meta-Evaluation
Mingqi Gao, Anthony Sicilia, Weiyan Shi
Across various non-verifiable tasks, human evaluation is reliable but expensive, while automatic metrics are more scalable but often biased. Building on prediction-powered inferenc…
Change My View? The Dynamics of Persuasion and Polarization in Online Discourse
David Freeborn, Malihe Alikani, Anthony Sicilia
Philosophical accounts of persuasion often assume that shared evidence and rational argumentation should lead to a convergence of views between peers, yet everyday discourse often…
MixDPO: Modeling Preference Strength for Pluralistic Alignment
Saki Imai, Pedram Heydari, Anthony Sicilia +3
Preference based alignment objectives implicitly assume that all human preferences are expressed with equal strength. In practice, however, preference strength varies across indivi…
Identifying & Interactively Refining Ambiguous User Goals for Data Visualization Code Generation
Mert İnan, Anthony Sicilia, Alex Xie +3
Establishing shared goals is a fundamental step in human-AI communication. However, ambiguities can lead to outputs that seem correct but fail to reflect the speaker's intent. In t…
Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity
Jiayi Zhang, Simon Yu, Derek Chong +4
Post-training alignment often reduces LLM diversity, leading to a phenomenon known as mode collapse. Unlike prior work that attributes this effect to algorithmic limitations, we id…
Measuring How (Not Just Whether) VLMs Build Common Ground
Saki Imai, Mert İnan, Anthony Sicilia +1
Large vision language models (VLMs) increasingly claim reasoning skills, yet current benchmarks evaluate them in single-turn or question answering settings. However, grounding is a…
SiLVERScore: Semantically-Aware Embeddings for Sign Language Generation Evaluation
Saki Imai, Mert İnan, Anthony Sicilia +1
Evaluating sign language generation is often done through back-translation, where generated signs are first recognized back to text and then compared to a reference using text-base…
BASIL: Bayesian Assessment of Sycophancy in LLMs
Katherine Atwell, Pedram Heydari, Anthony Sicilia +1
Sycophancy (overly agreeable or flattering behavior) poses a fundamental challenge for human-AI collaboration, particularly in high-stakes decision-making domains such as health, l…
Better Slow than Sorry: Introducing Positive Friction for Reliable Dialogue Systems
Mert İnan, Anthony Sicilia, Suvodip Dey +6
While theories of discourse and cognitive science have long recognized the value of unhurried pacing, recent dialogue research tends to minimize friction in conversational systems.…
Contextual ASR Error Handling with LLMs Augmentation for Goal-Oriented Conversational AI
Yuya Asano, Sabit Hassan, Paras Sharma +4
General-purpose automatic speech recognition (ASR) systems do not always perform well in goal-oriented dialogue. Existing ASR correction methods rely on prior user data or named en…
Accounting for Sycophancy in Language Model Uncertainty Estimation
Anthony Sicilia, Mert Inan, Malihe Alikhani
Effective human-machine collaboration requires machine learning models to externalize uncertainty, so users can reflect and intervene when necessary. For language models, these rep…
Eliciting Uncertainty in Chain-of-Thought to Mitigate Bias against Forecasting Harmful User Behaviors
Anthony Sicilia, Malihe Alikhani
Conversation forecasting tasks a model with predicting the outcome of an unfolding conversation. For instance, it can be applied in social media moderation to predict harmful user…
Generating Signed Language Instructions in Large-Scale Dialogue Systems
Mert İnan, Katherine Atwell, Anthony Sicilia +2
We introduce a goal-oriented conversational AI system enhanced with American Sign Language (ASL) instructions, presenting the first implementation of such a system on a worldwide m…
Active Learning for Robust and Representative LLM Generation in Safety-Critical Scenarios
Sabit Hassan, Anthony Sicilia, Malihe Alikhani
Ensuring robust safety measures across a wide range of scenarios is crucial for user-facing systems. While Large Language Models (LLMs) can generate valuable data for safety measur…
An Active Learning Framework for Inclusive Generation by Large Language Models
Sabit Hassan, Anthony Sicilia, Malihe Alikhani
Ensuring that Large Language Models (LLMs) generate text representative of diverse sub-populations is essential, particularly when key concepts related to under-represented groups…
Evaluating Theory of (an uncertain) Mind: Predicting the Uncertain Beliefs of Others in Conversation Forecasting
Anthony Sicilia, Malihe Alikhani
Typically, when evaluating Theory of Mind, we consider the beliefs of others to be binary: held or not held. But what if someone is unsure about their own beliefs? How can we quant…
Deal, or no deal (or who knows)? Forecasting Uncertainty in Conversations using Large Language Models
Anthony Sicilia, Hyunwoo Kim, Khyathi Raghavi Chandu +2
Effective interlocutors account for the uncertain goals, beliefs, and emotions of others. But even the best human conversationalist cannot perfectly anticipate the trajectory of a…
Learning to Generate Equitable Text in Dialogue from Biased Training Data
Anthony Sicilia, Malihe Alikhani
The ingrained principles of fairness in a dialogue system's decision-making process and generated responses are crucial for user engagement, satisfaction, and task achievement. Abs…
HumBEL: A Human-in-the-Loop Approach for Evaluating Demographic Factors of Language Models in Human-Machine Conversations
Anthony Sicilia, Jennifer C. Gates, Malihe Alikhani
While demographic factors like age and gender change the way people talk, and in particular, the way people talk to machines, there is little investigation into how large pre-train…
LEATHER: A Framework for Learning to Generate Human-like Text in Dialogue
Anthony Sicilia, Malihe Alikhani
Algorithms for text-generation in dialogue can be misguided. For example, in task-oriented settings, reinforcement learning that optimizes only task-success can lead to abysmal lex…
Modeling Non-Cooperative Dialogue: Theoretical and Empirical Insights
Anthony Sicilia, Tristan Maidment, Pat Healy +1
Investigating cooperativity of interlocutors is central in studying pragmatics of dialogue. Models of conversation that only assume cooperative agents fail to explain the dynamics…
PAC-Bayesian Domain Adaptation Bounds for Multiclass Learners
Anthony Sicilia, Katherine Atwell, Malihe Alikhani +1
Multiclass neural networks are a common tool in modern unsupervised domain adaptation, yet an appropriate theoretical description for their non-uniform sample complexity is lacking…
Test-time Fourier Style Calibration for Domain Generalization
Xingchen Zhao, Chang Liu, Anthony Sicilia +2
The topic of generalizing machine learning models learned on a collection of source domains to unknown target domains is challenging. While many domain generalization (DG) methods…
The Change that Matters in Discourse Parsing: Estimating the Impact of Domain Shift on Parser Error
Katherine Atwell, Anthony Sicilia, Seong Jae Hwang +1
Discourse analysis allows us to attain inferences of a text document that extend beyond the sentence-level. The current performance of discourse models is very low on texts outside…