13 citations · 16 across the 8 of their papers we have counts for
9 papers · 1 filter
No More Free Lunch: Corpus Task Complexity Matters as Corpora Grow
Prasann Singhal, Amanda Bertsch, Jacob Steinhardt +1
Given a large corpus, the questions one might ask can vary -- from "When was the first human heart transplant?" to "What are all the contradictory claims in this literature?" -- bu…
Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale
Siddharth Gollapudi, Nilesh Gupta, Prasann Singhal +1
Language models (LMs) raise an intriguing alternative to vector-based retrieval: conditioning on an in-context corpus and directly generating a relevant answer. However, prior work…
Assessing Robustness to Spurious Correlations in Post-Training Language Models
Julia Shuieh, Prasann Singhal, Apaar Shanker +3
Supervised and preference-based fine-tuning techniques have become popular for aligning large language models (LLMs) with user intent and correctness criteria. However, real-world…
ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models
Liyan Tang, Grace Kim, Xinyu Zhao +12
Chart understanding presents a unique challenge for large vision-language models (LVLMs), as it requires the integration of sophisticated textual and visual reasoning capabilities.…
To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning
Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez +7
Chain-of-thought (CoT) via prompting is the de facto method for eliciting reasoning capabilities from large language models (LLMs). But for what kinds of tasks is this extra ``thin…
D2PO: Discriminator-Guided DPO with Response Evaluation Models
Prasann Singhal, Nathan Lambert, Scott Niekum +2
Varied approaches for aligning language models have been proposed, including supervised fine-tuning, RLHF, and direct optimization methods such as DPO. Although DPO has rapidly gai…
A Long Way to Go: Investigating Length Correlations in RLHF
Prasann Singhal, Tanya Goyal, Jiacheng Xu +1
Great success has been reported using Reinforcement Learning from Human Feedback (RLHF) to align large language models, with open preference datasets enabling wider experimentation…
EEL: Efficiently Encoding Lattices for Reranking
Prasann Singhal, Jiacheng Xu, Xi Ye +1
Standard decoding approaches for conditional text generation tasks typically search for an output hypothesis with high model probability, but this may not yield the best hypothesis…
Assessing Out-of-Domain Language Model Performance from Few Examples
Prasann Singhal, Jarad Forristal, Xi Ye +1
While pretrained language models have exhibited impressive generalization capabilities, they still behave unpredictably under certain domain shifts. In particular, a model may lear…