626 citations · 1.7k across the 87 of their papers we have counts for
14 papers · 1 filter
EditLens: Quantifying the Extent of AI Editing in Text
Katherine Thai, Bradley Emi, Elyas Masrour +1
A significant proportion of queries to large language models ask them to edit user-provided text, rather than generate new text from scratch. While previous work focuses on detecti…
AI use in American newspapers is widespread, uneven, and rarely disclosed
Jenna Russell, Marzena Karpinska, Destiny Akinode +4
AI is rapidly transforming journalism, but the extent of its use in published newspaper articles remains unclear. We address this gap by auditing a large-scale dataset of 186K arti…
Literary Evidence Retrieval via Long-Context Language Models
Katherine Thai, Mohit Iyyer
How well do modern long-context language models understand literary fiction? We explore this question via the task of literary evidence retrieval, repurposing the RELiC dataset of…
OWL: Probing Cross-Lingual Recall of Memorized Texts via World Literature
Alisha Srivastava, Emir Korukluoglu, Minh Nhat Le +4
Large language models (LLMs) are known to memorize and recall English text from their pretraining data. However, the extent to which this ability generalizes to non-English languag…
Does quantization affect models' performance on long-context tasks?
Anmol Mekala, Anirudh Atmakuru, Yixiao Song +2
Large language models (LLMs) now support context windows exceeding 128K tokens, but this comes with significant memory requirements and high inference latency. Quantization can mit…
Frankentext: Stitching random text fragments into long-form narratives
Chau Minh Pham, Jenna Russell, Dzung Pham +1
We introduce Frankentexts, a long-form narrative generation paradigm that treats an LLM as a composer of existing texts rather than as an author. Given a writing prompt and thousan…
VeriFastScore: Speeding up long-form factuality evaluation
Rishanth Rajendhran, Amir Zadeh, Matthew Sarte +2
Metrics like FactScore and VeriScore that evaluate long-form factuality operate by decomposing an input response into atomic claims and then individually verifying each claim. Whil…
BLEUBERI: BLEU is a surprisingly effective reward for instruction following
Yapei Chang, Yekyung Kim, Michael Krumdick +4
Reward models are central to aligning LLMs with human preferences, but they are costly to train, requiring large-scale human-labeled preference data and powerful pretrained LLM bac…
BEARCUBS: A benchmark for computer-using web agents
Yixiao Song, Katherine Thai, Chau Minh Pham +3
Modern web agents possess computer use abilities that allow them to interact with webpages by sending commands to a virtual keyboard and mouse. While such agents have considerable…
One ruler to measure them all: Benchmarking multilingual long-context language models
Yekyung Kim, Jenna Russell, Marzena Karpinska +1
We present ONERULER, a multilingual benchmark designed to evaluate long-context language models across 26 languages. ONERULER adapts the English-only RULER benchmark (Hsieh et al.,…
CLIPPER: Compression enables long-context synthetic data generation
Chau Minh Pham, Yapei Chang, Mohit Iyyer
LLM developers are increasingly reliant on synthetic data, but generating high-quality data for complex long-context reasoning tasks remains challenging. We introduce CLIPPER, a co…
Whose story is it? Personalizing story generation by inferring author styles
Nischal Ashok Kumar, Chau Minh Pham, Mohit Iyyer +1
Personalization is critical for improving user experience in interactive writing and educational applications, yet remains understudied in story generation. We study the task of pe…
OverThink: Slowdown Attacks on Reasoning LLMs
Abhinav Kumar, Jaechul Roh, Ali Naseh +4
A reasoning language model (RLM) generates costly reasoning tokens, often hidden from the users, that help it excel at many tasks. Our Overthink attack targets RLM-based applicatio…
People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text
Jenna Russell, Marzena Karpinska, Mohit Iyyer
In this paper, we study how well humans can detect text generated by commercial LLMs (GPT-4o, Claude, o1). We hire annotators to read 300 non-fiction English articles, label them a…