4 papers · 1 filter
Challenges in Trustworthy Human Evaluation of Chatbots
Wenting Zhao, Alexander M. Rush, Tanya Goyal
Open community-driven platforms like Chatbot Arena that collect user preference data from site visitors have gained a reputation as one of the most trustworthy publicly available b…
Commit0: Library Generation from Scratch
Wenting Zhao, Nan Jiang, Celine Lee +4
With the goal of benchmarking generative systems beyond expert software development ability, we introduce Commit0, a benchmark that challenges AI agents to write libraries from scr…
Great Memory, Shallow Reasoning: Limits of NN-LMs
Shangyi Geng, Wenting Zhao, Alexander M Rush
-nearest neighbor language models (NN-LMs), which integrate retrieval with next-word prediction, have demonstrated strong performance in language modeling as well as downstre…
I Could've Asked That: Reformulating Unanswerable Questions
Wenting Zhao, Ge Gao, Claire Cardie +1
When seeking information from unfamiliar documents, users frequently pose questions that cannot be answered by the documents. While existing large language models (LLMs) identify t…