9 citations · 34 across the 71 of their papers we have counts for
32 papers · 1 filter
HLTCOE Evaluation Team at TREC 2025: VQA Track
Dengjia Zhang, Charles Weng, Katherine Guerrerio +5
The HLTCOE Evaluation team participated in TREC VQA's Answer Generation (AG) task, for which we developed a listwise learning framework that aims to improve semantic precision and…
All Claims Are Equal, but Some Claims Are More Equal Than Others: Importance-Sensitive Factuality Evaluation of LLM Generations
Miriam Wanner, Leif Azzopardi, Paul Thomas +3
Existing methods for evaluating the factuality of large language model (LLM) responses treat all claims as equally important. This results in misleading evaluations when vital info…
SocialNLI: A Dialogue-Centric Social Inference Dataset
Akhil Deo, Kate Sanders, Benjamin Van Durme
Making theory-of-mind inferences from human dialogue is a strong indicator of a model's underlying social abilities, which are fundamental for adept AI assistants. However, large l…
Seeing Through the MiRAGE: Evaluating Multimodal Retrieval Augmented Generation
Alexander Martin, William Walden, Reno Kriz +5
We introduce MiRAGE, an evaluation framework for retrieval-augmented generation (RAG) from multimodal sources. As audiovisual media becomes a more prevalent source of information o…
Sample-Efficient Online Learning in LM Agents via Hindsight Trajectory Rewriting
Michael Y. Hu, Benjamin Van Durme, Jacob Andreas +1
Language model (LM) agents deployed in novel environments often exhibit poor sample efficiency when learning from sequential interactions. This significantly hinders the usefulness…
The Alignment Waltz: Jointly Training Agents to Collaborate for Safety
Jingyu Zhang, Haozhu Wang, Eric Michael Smith +7
Harnessing the power of LLMs requires a delicate dance between being helpful and harmless. This creates a fundamental tension between two competing challenges: vulnerability to adv…
SEQR: Secure and Efficient QR-based LoRA Routing
William Fleshman, Benjamin Van Durme
Low-Rank Adaptation (LoRA) has become a standard technique for parameter-efficient fine-tuning of large language models, enabling large libraries of LoRAs, each for a specific task…
mmBERT: A Modern Multilingual Encoder with Annealed Language Learning
Marc Marone, Orion Weller, William Fleshman +3
Encoder-only languages models are frequently used for a variety of standard machine learning tasks, including classification and retrieval. However, there has been a lack of recent…
LoRA-Augmented Generation (LAG) for Knowledge-Intensive Language Tasks
William Fleshman, Benjamin Van Durme
The proliferation of fine-tuned language model experts for specific tasks and domains signals the need for efficient selection and combination methods. We propose LoRA-Augmented Ge…
MegaWika 2: A More Comprehensive Multilingual Collection of Articles and their Sources
Samuel Barham, Chandler May, Benjamin Van Durme
We introduce MegaWika 2, a large, multilingual dataset of Wikipedia articles with their citations and scraped web sources; articles are represented in a rich data structure, and sc…
Language Models and Logic Programs for Trustworthy Tax Reasoning
William Jurayj, Nils Holzenberger, Benjamin Van Durme
According to the United States Internal Revenue Service, ``the average American spends and 13 hours filing their taxes''. Even beyond the U.S., tax filing requires complex…
Seq vs Seq: An Open Suite of Paired Encoders and Decoders
Orion Weller, Kathryn Ricci, Marc Marone +3
The large language model (LLM) community focuses almost exclusively on decoder-only language models, since they are easier to use for text generation. However, a large subset of th…
Compactor: Calibrated Query-Agnostic KV Cache Compression with Approximate Leverage Scores
Vivek Chari, Benjamin Van Durme
Modern Large Language Models (LLMs) are increasingly trained to support very large context windows. We present Compactor, a training-free, query-agnostic KV compression strategy th…
How Grounded is Wikipedia? A Study on Structured Evidential Support and Retrieval
William Walden, Kathryn Ricci, Miriam Wanner +4
Wikipedia is a critical resource for modern NLP, serving as a rich repository of up-to-date and citation-backed information on a wide variety of subjects. The reliability of Wikipe…
Jailbreak Distillation: Renewable Safety Benchmarking
Jingyu Zhang, Ahmed Elgohary, Xiawei Wang +5
Large language models (LLMs) are rapidly deployed in critical applications, raising urgent needs for robust safety benchmarking. We propose Jailbreak Distillation (JBDistill), a no…
Rank-K: Test-Time Reasoning for Listwise Reranking
Eugene Yang, Andrew Yates, Kathryn Ricci +4
Retrieve-and-rerank is a popular retrieval pipeline because of its ability to make slow but effective rerankers efficient enough at query time by reducing the number of comparisons…
Always Tell Me The Odds: Fine-grained Conditional Probability Estimation
Liaoyaqi Wang, Zhengping Jiang, Anqi Liu +1
We present a state-of-the-art model for fine-grained probability estimation of propositions conditioned on context. Recent advances in large language models (LLMs) have significant…
MICE for CATs: Model-Internal Confidence Estimation for Calibrating Agents with Tools
Nishant Subramani, Jason Eisner, Justin Svegliato +3
Tool-using agents that act in the world need to be both useful and safe. Well-calibrated model confidences can be used to weigh the risk versus reward of potential actions, but pri…
Certified Mitigation of Worst-Case LLM Copyright Infringement
Jingyu Zhang, Jiacan Yu, Marc Marone +2
The exposure of large language models (LLMs) to copyrighted material during pre-training raises concerns about unintentional copyright infringement post deployment. This has driven…
Bonsai: Interpretable Tree-Adaptive Grounded Reasoning
Kate Sanders, Benjamin Van Durme
To develop general-purpose collaborative agents, humans need reliable AI systems that can (1) adapt to new domains and (2) transparently reason with uncertainty to allow for verifi…
arXiv2Table: Toward Realistic Benchmarking and Evaluation for LLM-Based Literature-Review Table Generation
Weiqi Wang, Jiefu Ou, Yangqiu Song +2
Literature review tables are essential for summarizing and comparing collections of scientific papers. In this paper, we study the automatic generation of such tables from a pool o…
SpectR: Dynamically Composing LM Experts with Spectral Routing
William Fleshman, Benjamin Van Durme
Training large, general-purpose language models poses significant challenges. The growing availability of specialized expert models, fine-tuned from pretrained models for specific…
WikiVideo: Article Generation from Multiple Videos
Alexander Martin, Reno Kriz, William Gantt Walden +5
We introduce the task of grounded article generation with the goal of creating a Wikipedia-style article from multiple diverse videos about real-world events -- from natural disast…
CLAIMCHECK: How Grounded are LLM Critiques of Scientific Papers?
Jiefu Ou, William Gantt Walden, Kate Sanders +13
A core part of scientific peer review involves providing expert critiques that directly assess the scientific claims a paper makes. While it is now possible to automatically genera…