2 citations · 8 across the 46 of their papers we have counts for
21 papers · 2 filters
Principled Context Engineering for RAG: Statistical Guarantees via Conformal Prediction
Debashish Chakraborty, Eugene Yang, Daniel Khashabi +2
Retrieval-Augmented Generation (RAG) enhances factual grounding in large language models (LLMs) by incorporating retrieved evidence, but LLM accuracy declines when long or noisy co…
CreativityPrism: A Cross-Domain Evaluation Framework for Large Language Model Creativity
Zhaoyi Joey Hou, Bowei Alvin Zhang, Yining Lu +9
Creativity is often seen as a hallmark of human intelligence. While large language models(LLMs) are increasingly perceived as generating creative text, there is still no cross-doma…
GOLD PANNING: Strategic Context Shuffling for Needle-in-Haystack Reasoning
Adam Byerly, Daniel Khashabi
Large language models (LLMs) exhibit pronounced position bias in long-context needle-in-haystack problems, systematically prioritizing the location of information over its relevanc…
The Alignment Waltz: Jointly Training Agents to Collaborate for Safety
Jingyu Zhang, Haozhu Wang, Eric Michael Smith +7
Harnessing the power of LLMs requires a delicate dance between being helpful and harmless. This creates a fundamental tension between two competing challenges: vulnerability to adv…
Challenging the Evaluator: LLM Sycophancy Under User Rebuttal
Sungwon Kim, Daniel Khashabi
Large Language Models (LLMs) often exhibit sycophancy, distorting responses to align with user beliefs, notably by readily agreeing with user counterarguments. Paradoxically, LLMs…
Jointly Reinforcing Diversity and Quality in Language Model Generations
Tianjian Li, Yiming Zhang, Ping Yu +5
Post-training of Large Language Models (LMs) often prioritizes accuracy and helpfulness at the expense of diversity. This creates a tension: while post-training improves response q…
The Flaw of Averages: Measuring Benchmark-Level Distributional Robustness
Arda Uzunoglu, Tianjian Li, Daniel Khashabi
Benchmarks are central to measuring progress in language models, but aggregate scores can obscure substantial variation across subdomains, making models appear broadly competent de…
Linguistic Nepotism: Trading-off Quality for Language Preference in Multilingual RAG
Dayeon Ki, Marine Carpuat, Paul McNamee +4
Multilingual Retrieval-Augmented Generation (mRAG) systems enable language models to answer knowledge-intensive queries with citation-supported responses across languages. Despite…
Evaluating the Evaluators: Are readability metrics good measures of readability?
Isabel Cachola, Daniel Khashabi, Mark Dredze
Plain Language Summarization (PLS) aims to distill complex documents into accessible summaries for non-expert audiences. In this paper, we conduct a thorough survey of PLS literatu…
Hell or High Water: Evaluating Agentic Recovery from External Failures
Andrew Wang, Sophia Hager, Adi Asija +2
As language model agents are applied to real world problems of increasing complexity, they will be expected to formulate plans across large search spaces. If those plans fail for r…
The Translation Barrier Hypothesis: Multilingual Generation with Large Language Models Suffers from Implicit Translation Failure
Niyati Bafna, Tianjian Li, Kenton Murray +4
Multilingual generation with large language models (LLMs) is often of poor quality for mid- to low-resource languages, but the causes for this are not well-understood. We first dem…
Feedback Friction: LLMs Struggle to Fully Incorporate External Feedback
Dongwei Jiang, Alvin Zhang, Andrew Wang +2
Recent studies have shown LLMs possess some ability to improve their responses when given external feedback. However, it remains unclear how effectively and thoroughly these models…
Jailbreak Distillation: Renewable Safety Benchmarking
Jingyu Zhang, Ahmed Elgohary, Xiawei Wang +5
Large language models (LLMs) are rapidly deployed in critical applications, raising urgent needs for robust safety benchmarking. We propose Jailbreak Distillation (JBDistill), a no…
SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning
Tianjian Li, Daniel Khashabi
Aligning language models with human preferences relies on pairwise preference datasets. While some studies suggest that on-policy data consistently outperforms off -policy data for…
BiomedSQL: Text-to-SQL for Scientific Reasoning on Biomedical Knowledge Bases
Mathew J. Koretsky, Maya Willey, Owen Bianchi +7
Biomedical researchers increasingly rely on large-scale structured databases for complex analytical tasks. However, current text-to-SQL systems often struggle to map qualitative sc…
Hidden in the Haystack: Smaller Needles are More Difficult for LLMs to Find
Owen Bianchi, Mathew J. Koretsky, Maya Willey +7
Large language models (LLMs) face significant challenges with needle-in-ahaystack tasks, where relevant information ("the needle") must be drawn from a large pool of irrelevant con…
Certified Mitigation of Worst-Case LLM Copyright Infringement
Jingyu Zhang, Jiacan Yu, Marc Marone +2
The exposure of large language models (LLMs) to copyrighted material during pre-training raises concerns about unintentional copyright infringement post deployment. This has driven…
ICL CIPHERS: Quantifying "Learning" in In-Context Learning via Substitution Ciphers
Zhouxiang Fang, Aayush Mishra, Muhan Gao +2
Recent works have suggested that In-Context Learning (ICL) operates in dual modes, i.e. task retrieval (remember learned patterns from pre-training) and task learning (inference-ti…
From Papers to Panoramas: Building Hierarchies of Scientific Literature at Scale
Muhan Gao, Jash Shah, Weiqi Wang +2
Scientific knowledge is growing rapidly, making it difficult to track progress and high-level conceptual links across broad disciplines. While tools like citation networks and sear…
arXiv2Table: Toward Realistic Benchmarking and Evaluation for LLM-Based Literature-Review Table Generation
Weiqi Wang, Jiefu Ou, Yangqiu Song +2
Literature review tables are essential for summarizing and comparing collections of scientific papers. In this paper, we study the automatic generation of such tables from a pool o…
CLAIMCHECK: How Grounded are LLM Critiques of Scientific Papers?
Jiefu Ou, William Gantt Walden, Kate Sanders +13
A core part of scientific peer review involves providing expert critiques that directly assess the scientific claims a paper makes. While it is now possible to automatically genera…