Publications (82)
Probing Biomedical Embeddings from Language Models
Qiao Jin, Bhuwan Dhingra, William W. Cohen +1
Contextualized word embeddings derived from pre-trained language models (LMs) show significant improvements on downstream NLP tasks. Pre-training on domain-specific corpora, such a…
Knowing When to Stop: Efficient Context Processing via Latent Sufficiency Signals
Roy Xie, Junlin Wang, Paul Rosu +4
Large language models (LLMs) process entire input contexts indiscriminately, which is inefficient when the information required to answer a query is localized within the context. W…
Characterizing the Efficiency vs. Accuracy Trade-off for Long-Context NLP Models
Phyllis Ang, Bhuwan Dhingra, Lisa Wu Wills
With many real-world applications of Natural Language Processing (NLP) comprising of long texts, there has been a rise in NLP benchmarks that measure the accuracy of models that ca…
Interface Design for Crowdsourcing Hierarchical Multi-Label Text Annotations
Rickard Stureborg, Bhuwan Dhingra, Jun Yang
Human data labeling is an important and expensive task at the heart of supervised learning systems. Hierarchies help humans understand and organize concepts. We ask whether and how…
Stratified Prediction-Powered Inference for Hybrid Language Model Evaluation
Adam Fisch, Joshua Maynez, R. Alex Hofer +3
Prediction-powered inference (PPI) is a method that improves statistical estimates based on limited human-labeled data. PPI achieves this by combining small amounts of human-labele…
Evaluating Explanations: How much do explanations from the teacher aid students?
Danish Pruthi, Rachit Bansal, Bhuwan Dhingra +5
While many methods purport to explain predictions by highlighting salient features, what aims these explanations serve and how they ought to be evaluated often go unstated. In this…
Bootstrapping Distantly Supervised IE using Joint Learning and Small Well-structured Corpora
Lidong Bing, Bhuwan Dhingra, Kathryn Mazaitis +2
We propose a framework to improve performance of distantly-supervised relation extraction, by jointly learning to solve two related tasks: concept-instance extraction and relation…
LensVLM: Selective Context Expansion for Compressed Visual Representation of Text
Roy Xie, Dan Friedman, Donghan Yu +7
Vision Language Models (VLMs) offer the exciting possibility of processing text as rendered images, bypassing the need for tokenizing the text into long token sequences. Since VLM…
Fuzzy Speculative Decoding for a Tunable Accuracy-Runtime Tradeoff
Maximilian Holsman, Yukun Huang, Bhuwan Dhingra
Speculative Decoding (SD) enforces strict distributional equivalence to the target model when accepting candidate tokens. While it maintains the target model's generation quality,…
Adversarial Math Word Problem Generation
Roy Xie, Chengxuan Huang, Junlin Wang +1
Large language models (LLMs) have significantly transformed the educational landscape. As current plagiarism detection tools struggle to keep pace with LLMs' rapid advancements, th…
IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representations
Deqing Fu, Ruohao Guo, Ghazal Khalighinejad +5
Current foundation models exhibit impressive capabilities when prompted either with text only or with both image and text inputs. But do their capabilities change depending on the…
Apple Intelligence Foundation Language Models: Tech Report 2025
Ethan Li, Anders Boesen Lindbo Larsen, Chen Zhang +395
We introduce two multilingual, multimodal foundation language models that power Apple Intelligence features across Apple devices and services: i a 3B-parameter on-device model opti…
Atomic Self-Consistency for Better Long Form Generations
Raghuveer Thirukovalluru, Yukun Huang, Bhuwan Dhingra
Recent work has aimed to improve LLM generations by filtering out hallucinations, thereby improving the precision of the information in responses. Correctness of a long-form respon…
Combating Adversarial Misspellings with Robust Word Recognition
Danish Pruthi, Bhuwan Dhingra, Zachary C. Lipton
To combat adversarial spelling mistakes, we propose placing a word recognition model in front of the downstream classifier. Our word recognition models build upon the RNN semi-char…
Hierarchical Multi-Instance Multi-Label Learning for Detecting Propaganda Techniques
Anni Chen, Bhuwan Dhingra
Since the introduction of the SemEval 2020 Task 11 (Martino et al., 2020a), several approaches have been proposed in the literature for classifying propaganda based on the rhetoric…
Weakly- and Semi-supervised Evidence Extraction
Danish Pruthi, Bhuwan Dhingra, Graham Neubig +1
For many prediction tasks, stakeholders desire not only predictions but also supporting evidence that a human can use to verify its correctness. However, in practice, additional an…
Calibrating Long-form Generations from Large Language Models
Yukun Huang, Yixin Liu, Raghuveer Thirukovalluru +2
To enhance Large Language Models' (LLMs) reliability, calibration is essential -- the model's assessed confidence scores should align with the actual likelihood of its responses be…
Extracting Polymer Nanocomposite Samples from Full-Length Documents
Ghazal Khalighinejad, Defne Circi, L. C. Brinson +1
This paper investigates the use of large language models (LLMs) for extracting sample lists of polymer nanocomposites (PNCs) from full-length materials science research papers. The…
How Much Backtracking is Enough? Exploring the Interplay of SFT and RL in Enhancing LLM Reasoning
Hongyi James Cai, Junlin Wang, Xiaoyin Chen +1
Recent advancements in large language models (LLMs) suggest that reinforcement learning (RL) effectively internalizes search strategies, yielding significant improvements on challe…
Text Generation with Exemplar-based Adaptive Decoding
Hao Peng, Ankur P. Parikh, Manaal Faruqui +2
We propose a novel conditioned text generation model. It draws inspiration from traditional template-based text generation techniques, where the source provides the content (i.e.,…
On the State of the Art in Authorship Attribution and Authorship Verification
Jacob Tyo, Bhuwan Dhingra, Zachary C. Lipton
Despite decades of research on authorship attribution (AA) and authorship verification (AV), inconsistent dataset splits/filtering and mismatched evaluation methods make it difficu…
Vision2Code: A Multi-Domain Benchmark for Evaluating Image-to-Code Generation
Ajay Vikram Periasami, Junlin Wang, Bhuwan Dhingra
Image-to-code generation tests whether a vision-language model (VLM) can recover the structure of an image enough to express it as executable code. Existing benchmarks either focus…
Differentiable Reasoning over a Virtual Knowledge Base
Bhuwan Dhingra, Manzil Zaheer, Vidhisha Balachandran +3
We consider the task of answering complex multi-hop questions using a corpus as a virtual knowledge base (KB). In particular, we describe a neural module, DrKIT, that traverses tex…
Evaluating Morphological Compositional Generalization in Large Language Models
Mete Ismayilzada, Defne Circi, Jonne Sälevä +6
Large language models (LLMs) have demonstrated significant progress in various natural language generation and understanding tasks. However, their linguistic generalization capabil…
Time-Aware Language Models as Temporal Knowledge Bases
Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos +3
Many facts come with an expiration date, from the name of the President to the basketball team Lebron James plays for. But language models (LMs) are trained on snapshots of data co…
Question Answering from Unstructured Text by Retrieval and Comprehension
Yusuke Watanabe, Bhuwan Dhingra, Ruslan Salakhutdinov
Open domain Question Answering (QA) systems must interact with external knowledge sources, such as web pages, to find relevant information. Information sources like Wikipedia, howe…
Atomic Consistency Preference Optimization for Long-Form Question Answering
Jingfeng Chen, Raghuveer Thirukovalluru, Junlin Wang +2
Large Language Models (LLMs) often produce factoid hallucinations - plausible yet incorrect answers. A common mitigation strategy is model alignment, which improves factual accurac…
Tweet2Vec: Character-Based Distributed Representations for Social Media
Bhuwan Dhingra, Zhong Zhou, Dylan Fitzpatrick +2
Text from social media provides a set of challenges that can cause traditional NLP approaches to fail. Informal language, spelling errors, abbreviations, and special characters are…
Interleaved Reasoning for Large Language Models via Reinforcement Learning
Roy Xie, David Qiu, Deepak Gopinath +5
Long chain-of-thought (CoT) significantly enhances the reasoning capabilities of large language models (LLMs). However, extensive reasoning traces lead to inefficiencies and increa…
Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text
Haitian Sun, Bhuwan Dhingra, Manzil Zaheer +3
Open Domain Question Answering (QA) is evolving from complex pipelined systems to end-to-end deep neural networks. Specialized neural models have been developed for extracting answ…
ReCaLL: Membership Inference via Relative Conditional Log-Likelihoods
Roy Xie, Junlin Wang, Ruomin Huang +5
The rapid scaling of large language models (LLMs) has raised concerns about the transparency and fair use of the data used in their pretraining. Detecting such content is challengi…
Learning to Deceive with Attention-Based Explanations
Danish Pruthi, Mansi Gupta, Bhuwan Dhingra +2
Attention mechanisms are ubiquitous components in neural architectures applied to natural language processing. In addition to yielding gains in predictive accuracy, attention weigh…
Document-as-Image Representations Fall Short for Scientific Retrieval
Ghazal Khalighinejad, Raghuveer Thirukovalluru, Alexander H. Oh +1
Many recent document embedding models are trained on document-as-image representations, embedding rendered pages as images rather than the underlying source. Meanwhile, existing be…
PRO-LONG: Programmatic Memory Enables Long-Horizon Reasoning
Alexis Fox, Junlin Wang, Paul Rosu +1
Long-horizon tasks require sustained perception, reasoning, and exploration, and are a persistent challenge for large language model (LLM) agents. This gap is reflected in their li…
Cite Pretrain: Retrieval-Free Knowledge Attribution for Large Language Models
Yukun Huang, Sanxing Chen, Jian Pei +2
Trustworthy language models should provide both correct and verifiable answers. However, citations generated directly by standalone LLMs are often unreliable. As a result, current…
Raccoon: Prompt Extraction Benchmark of LLM-Integrated Applications
Junlin Wang, Tianyi Yang, Roy Xie +1
With the proliferation of LLM-integrated applications such as GPT-s, millions are deployed, offering valuable services through proprietary instruction prompts. These systems, howev…
Selectively Answering Ambiguous Questions
Jeremy R. Cole, Michael J. Q. Zhang, Daniel Gillick +3
Trustworthy language models should abstain from answering questions when they do not know the answer. However, the answer to a question can be unknown for a variety of reasons. Pri…
PubMedQA: A Dataset for Biomedical Research Question Answering
Qiao Jin, Bhuwan Dhingra, Zhengping Liu +2
We introduce PubMedQA, a novel biomedical question answering (QA) dataset collected from PubMed abstracts. The task of PubMedQA is to answer research questions with yes/no/maybe (e…
ChatShop: Interactive Information Seeking with Language Agents
Sanxing Chen, Sam Wiseman, Bhuwan Dhingra
The desire and ability to seek new information strategically are fundamental to human learning but often overlooked in current language agent evaluation. We analyze a popular web s…
Linguistic Knowledge as Memory for Recurrent Neural Networks
Bhuwan Dhingra, Zhilin Yang, William W. Cohen +1
Training recurrent neural networks to model long term dependencies is difficult. Hence, we propose to use external linguistic knowledge as an explicit signal to inform the model wh…
Hierarchical Multi-Label Classification of Online Vaccine Concerns
Chloe Qinyu Zhu, Rickard Stureborg, Bhuwan Dhingra
Vaccine concerns are an ever-evolving target, and can shift quickly as seen during the COVID-19 pandemic. Identifying longitudinal trends in vaccine concerns and misinformation mig…
Simple and Effective Semi-Supervised Question Answering
Bhuwan Dhingra, Danish Pruthi, Dheeraj Rajagopal
Recent success of deep learning models for the task of extractive Question Answering (QA) is hinged on the availability of large annotated corpora. However, large domain specific a…
LDDR: Linear-DPP-Based Dynamic-Resolution Frame Sampling for Video MLLMs
Jingfeng Chen, Jiawen Qian, Wendi Deng +5
Video understanding in multimodal large language models requires selecting informative frames from long, redundant videos under limited visual-token budgets. Existing methods often…
Improving Model Alignment Through Collective Intelligence of Open-Source LLMS
Junlin Wang, Roy Xie, Shang Zhu +6
Building helpful and harmless large language models (LLMs) requires effective model alignment approach based on human instructions and feedback, which necessitates high-quality hum…
GLoMo: Unsupervisedly Learned Relational Graphs as Transferable Representations
Zhilin Yang, Jake Zhao, Bhuwan Dhingra +4
Modern deep transfer learning approaches have mainly focused on learning generic feature vectors from one task that are transferable to other tasks, such as word embeddings in lang…
Learning the Legibility of Visual Text Perturbations
Dev Seth, Rickard Stureborg, Danish Pruthi +1
Many adversarial attacks in NLP perturb inputs to produce visually similar strings ('ergo' 'rgo') which are legible to humans but degrade model performance. Altho…
Towards End-to-End Reinforcement Learning of Dialogue Agents for Information Access
Bhuwan Dhingra, Lihong Li, Xiujun Li +4
This paper proposes KB-InfoBot -- a multi-turn dialogue agent which helps users search Knowledge Bases (KBs) without composing complicated queries. Such goal-oriented dialogue agen…
Bayesian Prediction-Powered Inference
R. Alex Hofer, Joshua Maynez, Bhuwan Dhingra +3
Prediction-powered inference (PPI) is a method that improves statistical estimates based on limited human-labeled data. Specifically, PPI methods provide tighter confidence interva…
A Platform for Investigating Public Health Content with Efficient Concern Classification
Christopher Li, Rickard Stureborg, Bhuwan Dhingra +1
A recent rise in online content expressing concerns with public health initiatives has contributed to already stalled uptake of preemptive measures globally. Future public health e…
Neural Models for Reasoning over Multiple Mentions using Coreference
Bhuwan Dhingra, Qiao Jin, Zhilin Yang +2
Many problems in NLP require aggregating information from multiple mentions of the same entity which may be far apart in the text. Existing Recurrent Neural Network (RNN) layers ar…
Words or Characters? Fine-grained Gating for Reading Comprehension
Zhilin Yang, Bhuwan Dhingra, Ye Yuan +3
Previous work combines word-level and character-level representations using concatenation or scalar weighting, which is suboptimal for high-level tasks like reading comprehension.…
RVPO: Risk-Sensitive Alignment via Variance Regularization
Ivan Montero, Tomasz Jurczyk, Bhuwan Dhingra
Current critic-less RLHF methods aggregate multi-objective rewards via an arithmetic mean, leaving them vulnerable to constraint neglect: high-magnitude success in one objective ca…
GenEOL: Harnessing the Generative Power of LLMs for Training-Free Sentence Embeddings
Raghuveer Thirukovalluru, Bhuwan Dhingra
Training-free embedding methods directly leverage pretrained large language models (LLMs) to embed text, bypassing the costly and complex procedure of contrastive learning. Previou…
Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Mining
Raghuveer Thirukovalluru, Rui Meng, Ye Liu +7
Contrastive learning (CL) is a prevalent technique for training embedding models, which pulls semantically similar examples (positives) closer in the representation space while pus…
Rethinking Inference-Time Scaling: Efficiency Limits and Linguistic Signals
Junlin Wang, Shang Zhu, Jon Saad-Falcon +7
There is intense interest in investigating how inference time compute (ITC) (e.g. repeated sampling, refinements, etc) can improve large language model (LLM) capabilities. While br…
Salient Span Masking for Temporal Understanding
Jeremy R. Cole, Aditi Chaudhary, Bhuwan Dhingra +1
Salient Span Masking (SSM) has shown itself to be an effective strategy to improve closed-book question answering performance. SSM extends general masked language model pretraining…
Real-time Factuality Assessment from Adversarial Feedback
Sanxing Chen, Yukun Huang, Bhuwan Dhingra
We show that existing evaluations for assessing the factuality of news from conventional sources, such as claims on fact-checking websites, result in high accuracies over time for…
ASQA: Factoid Questions Meet Long-Form Answers
Ivan Stelmakh, Yi Luan, Bhuwan Dhingra +1
An abundance of datasets and availability of reliable evaluation metrics have resulted in strong progress in factoid question answering (QA). This progress, however, does not easil…
When Greedy Wins: Emergent Exploitation Bias in Meta-Bandit LLM Training
Sanxing Chen, Xiaoyin Chen, Yukun Huang +2
While Large Language Models (LLMs) hold promise to become autonomous agents, they often explore suboptimally in sequential decision-making. Recent work has sought to enhance this c…
Generalizability of Large Language Model-Based Agents: A Comprehensive Survey
Minxing Zhang, Yi Yang, Roy Xie +3
Large Language Model (LLM)-based agents have emerged as a new paradigm that extends LLMs' capabilities beyond text generation to dynamic interaction with external environments. By…
A Comparative Study of Word Embeddings for Reading Comprehension
Bhuwan Dhingra, Hanxiao Liu, Ruslan Salakhutdinov +1
The focus of past machine learning research for Reading Comprehension tasks has been primarily on the design of novel deep learning architectures. Here we show that seemingly minor…
Automated Benchmark Auditing for AI Agents and Large Language Models
Junlin Wang, Federico Bianchi, Shang Zhu +4
Modern AI benchmarks operate at a complexity that outpaces traditional verification methods. Tasks authored by domain experts often contain implicit assumptions, incomplete environ…
Embedding Text in Hyperbolic Spaces
Bhuwan Dhingra, Christopher J. Shallue, Mohammad Norouzi +2
Natural language text exhibits hierarchical structure in a variety of respects. Ideally, we could incorporate our prior knowledge of this hierarchical structure into unsupervised l…
A User Simulator for Task-Completion Dialogues
Xiujun Li, Zachary C. Lipton, Bhuwan Dhingra +3
Despite widespread interests in reinforcement-learning for task-oriented dialogue systems, several obstacles can frustrate research and development progress. First, reinforcement l…
Using Graphs of Classifiers to Impose Declarative Constraints on Semi-supervised Learning
Lidong Bing, William W. Cohen, Bhuwan Dhingra
We propose a general approach to modeling semi-supervised learning (SSL) algorithms. Specifically, we present a declarative language for modeling both traditional supervised classi…
Coding Agents are Effective Long-Context Processors
Weili Cao, Xunjian Yin, Bhuwan Dhingra +1
Large Language Models (LLMs) have demonstrated remarkable progress in scaling to access massive contexts. However, the access is via the latent and uninterpretable attention mechan…
Over-Searching in Search-Augmented Large Language Models
Roy Xie, Deepak Gopinath, David Qiu +4
Search-augmented large language models (LLMs) excel at knowledge-intensive tasks by integrating external retrieval. However, they often over-search -- unnecessarily invoking search…
Fool Me Twice: Entailment from Wikipedia Gamification
Julian Martin Eisenschlos, Bhuwan Dhingra, Jannis Bulian +2
We release FoolMeTwice (FM2 for short), a large dataset of challenging entailment pairs collected through a fun multi-player game. Gamification encourages adversarial examples, dra…
Differentiable Open-Ended Commonsense Reasoning
Bill Yuchen Lin, Haitian Sun, Bhuwan Dhingra +3
Current commonsense reasoning research focuses on developing models that use commonsense knowledge to answer multiple-choice questions. However, systems designed to answer multiple…
Reasoning Over Virtual Knowledge Bases With Open Predicate Relations
Haitian Sun, Pat Verga, Bhuwan Dhingra +2
We present the Open Predicate Query Language (OPQL); a method for constructing a virtual KB (VKB) trained entirely from text. Large Knowledge Bases (KBs) are indispensable for a wi…
MatViX: Multimodal Information Extraction from Visually Rich Articles
Ghazal Khalighinejad, Sharon Scott, Ollie Liu +4
Multimodal information extraction (MIE) is crucial for scientific literature, where valuable data is often spread across text, figures, and tables. In materials science, extracting…
Quasar: Datasets for Question Answering by Search and Reading
Bhuwan Dhingra, Kathryn Mazaitis, William W. Cohen
We present two new large-scale datasets aimed at evaluating systems designed to comprehend a natural language query and extract its answer from a large corpus of text. The Quasar-S…
Handling Divergent Reference Texts when Evaluating Table-to-Text Generation
Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh +3
Automatically constructed datasets for generating text from semi-structured data (tables), such as WikiBio, often contain reference texts that diverge from the information in the c…
Staircase Streaming for Low-Latency Multi-Agent Inference
Junlin Wang, Jue Wang, Zhen +5
Recent advances in large language models (LLMs) opened up new directions for leveraging the collective expertise of multiple LLMs. These methods, such as Mixture-of-Agents, typical…
DIFFQG: Generating Questions to Summarize Factual Changes
Jeremy R. Cole, Palak Jain, Julian Martin Eisenschlos +3
Identifying the difference between two versions of the same article is useful to update knowledge bases and to understand how articles evolve. Paired texts occur naturally in diver…
Text-Guided Semantic Image Encoder
Raghuveer Thirukovalluru, Xiaochuang Han, Bhuwan Dhingra +2
Image encoders, a fundamental component of vision-language models (VLMs), are typically pretrained independently before being aligned with a language model. This standard paradigm…
InData: Towards Secure Multi-Step, Tool-Based Data Analysis
Karthikeyan K, Raghuveer Thirukovalluru, Bhuwan Dhingra +1
Large language model agents for data analysis typically generate and execute code directly on databases. However, when applied to sensitive data, this approach poses significant se…
Tailoring Vaccine Messaging with Common-Ground Opinions
Rickard Stureborg, Sanxing Chen, Ruoyu Xie +6
One way to personalize chatbot interactions is by establishing common ground with the intended reader. A domain where establishing mutual understanding could be particularly impact…
ToTTo: A Controlled Table-To-Text Generation Dataset
Ankur P. Parikh, Xuezhi Wang, Sebastian Gehrmann +4
We present ToTTo, an open-domain English table-to-text dataset with over 120,000 training examples that proposes a controlled generation task: given a Wikipedia table and a set of…
DeepFact: Co-Evolving Benchmarks and Agents for Deep Research Factuality
Yukun Huang, Leonardo F. R. Ribeiro, Momchil Hardalov +3
Search-augmented LLM agents can produce deep research reports (DRRs), but verifying claim-level factuality remains challenging. Existing fact-checkers are primarily designed for ge…
To Trust or Not to Trust? Enhancing Large Language Models' Situated Faithfulness to External Contexts
Yukun Huang, Sanxing Chen, Hongyi Cai +1
Large Language Models (LLMs) are often augmented with external contexts, such as those used in retrieval-augmented generation (RAG). However, these contexts can be inaccurate or in…
Gated-Attention Readers for Text Comprehension
Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang +2
In this paper we study the problem of answering cloze-style questions over documents. Our model, the Gated-Attention (GA) Reader, integrates a multi-hop architecture with a novel a…