18 citations · 30 across the 14 of their papers we have counts for
16 papers
Insider Knowledge: How Much Can RAG Systems Gain from Evaluation Secrets?
Laura Dietz, Bryan Li, Eugene Yang +3
RAG systems are increasingly evaluated and optimized using LLM judges, an approach that is rapidly becoming the dominant paradigm for system assessment. Nugget-based approaches in…
Auto-ARGUE: LLM-Based Report Generation Evaluation
William Walden, Marc Mason, Orion Weller +10
Generation of citation-backed reports is a primary use case for retrieval-augmented generation (RAG) systems. While open-source evaluation tools exist for various RAG tasks, tools…
Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale
Bowen Jiang, Zhuoqun Hao, Young-Min Cho +6
Large Language Models (LLMs) have emerged as personalized assistants for users across a wide range of tasks -- from offering writing support to delivering tailored recommendations…
Leveraging Domain Knowledge at Inference Time for LLM Translation: Retrieval versus Generation
Bryan Li, Jiaming Luo, Eleftheria Briakou +1
While large language models (LLMs) have been increasingly adopted for machine translation (MT), their performance for specialist domains such as medicine and law remains an open ch…
Multilingual Retrieval Augmented Generation for Culturally-Sensitive Tasks: A Benchmark for Cross-lingual Robustness
Bryan Li, Fiona Luo, Samar Haider +7
The paradigm of retrieval-augmented generated (RAG) helps mitigate hallucinations of large language models (LLMs). However, RAG also introduces biases contained within the retrieve…
Eliciting Better Multilingual Structured Reasoning from LLMs through Code
Bryan Li, Tamer Alkhouli, Daniele Bonadiman +2
The development of large language models (LLM) has shown progress on reasoning, though studies have largely considered either English or simple reasoning tasks. To address this, we…
Large Language Models as Sous Chefs: Revising Recipes with GPT-3
Alyssa Hwang, Bryan Li, Zhaoyi Hou +1
With their remarkably improved text generation and prompting capabilities, large language models can adapt existing written information into forms that are easier to use and unders…
This Land is {Your, My} Land: Evaluating Geopolitical Biases in Language Models
Bryan Li, Samar Haider, Chris Callison-Burch
Do the Spratly Islands belong to China, the Philippines, or Vietnam? A pretrained large language model (LLM) may answer differently if asked in the languages of each claimant count…
PAXQA: Generating Cross-lingual Question Answering Examples at Training Scale
Bryan Li, Chris Callison-Burch
Existing question answering (QA) systems owe much of their success to large, high-quality training data. Such annotation efforts are costly, and the difficulty compounds in the cro…
CREATIVESUMM: Shared Task on Automatic Summarization for Creative Writing
Divyansh Agarwal, Alexander R. Fabbri, Simeng Han +7
This paper introduces the shared task of summarizing documents in several creative domains, namely literary texts, movie scripts, and television scripts. Summarizing these creative…
Word Alignment in the Era of Deep Learning: A Tutorial
Bryan Li
The word alignment task, despite its prominence in the era of statistical machine translation (SMT), is niche and under-explored today. In this two-part tutorial, we argue for the…
Bidirectional Language Models Are Also Few-shot Learners
Ajay Patel, Bryan Li, Mohammad Sadegh Rasooli +3
Large language models such as GPT-3 (Brown et al., 2020) can perform arbitrary tasks without undergoing fine-tuning after being prompted with only a few labeled examples. An arbitr…
Multilingual Bidirectional Unsupervised Translation Through Multilingual Finetuning and Back-Translation
Bryan Li, Mohammad Sadegh Rasooli, Ajay Patel +1
We propose a two-stage approach for training a single NMT model to translate unseen languages both to and from English. For the first stage, we initialize an encoder-decoder model…
: A Simplified Commonsense Inference Evaluation for Story Prose
Bryan Li, Lara J. Martin, Chris Callison-Burch
Transformers have been showing near-human performance on a variety of tasks, but they are not without their limitations. We discuss the issue of conflating results of transformers…
Exploring Content Selection in Summarization of Novel Chapters
Faisal Ladhak, Bryan Li, Yaser Al-Onaizan +1
We present a new summarization task, generating summaries of novel chapters using summary/chapter pairs from online study guides. This is a harder task than the news summarization…
Cantonese Automatic Speech Recognition Using Transfer Learning from Mandarin
Bryan Li, Xinyue Wang, Homayoon Beigi
We propose a system to develop a basic automatic speech recognizer(ASR) for Cantonese, a low-resource language, through transfer learning of Mandarin, a high-resource language. We…