91 citations · 122 across the 21 of their papers we have counts for
23 papers
IdeaLens: Detecting AI Ideas in Long-form Writing
Rishanth Rajendhran, Minjoon Choi, Jenna Russell +5
While modern AI detectors identify who wrote the words, emerging policies on AI use increasingly hinge on a different question: who came up with the ideas? We introduce IdeaLens, a…
AI translation of literary texts is "fine", but readers still prefer human translations
Yves Ferstler, Adam Podoxin, Ty Brassington +5
AI translation of literary works is increasingly common. While the content may be rendered adequately, we do not know enough about how readers experience it in terms of immersivene…
AI use in American newspapers is widespread, uneven, and rarely disclosed
Jenna Russell, Marzena Karpinska, Destiny Akinode +4
AI is rapidly transforming journalism, but the extent of its use in published newspaper articles remains unclear. We address this gap by auditing a large-scale dataset of 186K arti…
Preliminary Ranking of WMT25 General Machine Translation Systems
Tom Kocmi, Eleftherios Avramidis, Rachel Bawden +25
We present the preliminary rankings of machine translation (MT) systems submitted to the WMT25 General Machine Translation Shared Task, as determined by automatic evaluation metric…
An Interdisciplinary Approach to Human-Centered Machine Translation
Marine Carpuat, Omri Asscher, Kalika Bali +17
Machine Translation (MT) tools are widely used today, often in contexts where professional translators are not present. Despite progress in MT technology, a gap persists between sy…
OWL: Probing Cross-Lingual Recall of Memorized Texts via World Literature
Alisha Srivastava, Emir Korukluoglu, Minh Nhat Le +4
Large language models (LLMs) are known to memorize and recall English text from their pretraining data. However, the extent to which this ability generalizes to non-English languag…
Does quantization affect models' performance on long-context tasks?
Anmol Mekala, Anirudh Atmakuru, Yixiao Song +2
Large language models (LLMs) now support context windows exceeding 128K tokens, but this comes with significant memory requirements and high inference latency. Quantization can mit…
One ruler to measure them all: Benchmarking multilingual long-context language models
Yekyung Kim, Jenna Russell, Marzena Karpinska +1
We present ONERULER, a multilingual benchmark designed to evaluate long-context language models across 26 languages. ONERULER adapts the English-only RULER benchmark (Hsieh et al.,…
OverThink: Slowdown Attacks on Reasoning LLMs
Abhinav Kumar, Jaechul Roh, Ali Naseh +4
A reasoning language model (RLM) generates costly reasoning tokens, often hidden from the users, that help it excel at many tasks. Our Overthink attack targets RLM-based applicatio…
People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text
Jenna Russell, Marzena Karpinska, Mohit Iyyer
In this paper, we study how well humans can detect text generated by commercial LLMs (GPT-4o, Claude, o1). We hire annotators to read 300 non-fiction English articles, label them a…
Preliminary WMT24 Ranking of General MT Systems and LLMs
Tom Kocmi, Eleftherios Avramidis, Rachel Bawden +18
This is the preliminary ranking of WMT24 General MT systems based on automatic metrics. The official ranking will be a human evaluation, which is superior to the automatic ranking…
CaLMQA: Exploring culturally specific long-form question answering across 23 languages
Shane Arora, Marzena Karpinska, Hung-Ting Chen +3
Despite rising global usage of large language models (LLMs), their ability to generate long-form answers to culturally specific questions remains unexplored in many languages. To f…
One Thousand and One Pairs: A "novel" challenge for long-context language models
Marzena Karpinska, Katherine Thai, Kyle Lo +2
Synthetic long-context LLM benchmarks (e.g., "needle-in-the-haystack") test only surface-level retrieval capabilities, but how well can long-context LLMs retrieve, synthesize, and…
Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation
Tom Kocmi, Vilém Zouhar, Eleftherios Avramidis +5
High-quality Machine Translation (MT) evaluation relies heavily on human judgments. Comprehensive error classification methods, such as Multidimensional Quality Metrics (MQM), are…
FABLES: Evaluating faithfulness and content selection in book-length summarization
Yekyung Kim, Yapei Chang, Marzena Karpinska +5
While long-context large language models (LLMs) can technically summarize book-length documents (>100K tokens), the length and complexity of the documents have so far prohibited ev…
Aurora-M: Open Source Continual Pre-training for Multilingual Language and Code
Taishi Nakamura, Mayank Mishra, Simone Tedeschi +42
Pretrained language models are an integral part of AI applications, but their high computational cost for training limits accessibility. Initiatives such as Bloom and StarCoder aim…
Large language models effectively leverage document-level context for literary translation, but critical errors persist
Marzena Karpinska, Mohit Iyyer
Large language models (LLMs) are competitive with the state of the art on a wide range of sentence-level translation datasets. However, their ability to translate paragraphs and do…
Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense
Kalpesh Krishna, Yixiao Song, Marzena Karpinska +2
The rise in malicious usage of large language models, such as fake content creation and academic plagiarism, has motivated the development of approaches that identify AI-generated…
Exploring Document-Level Literary Machine Translation with Parallel Paragraphs from World Literature
Katherine Thai, Marzena Karpinska, Kalpesh Krishna +4
Literary translation is a culturally significant task, but it is bottlenecked by the small number of qualified literary translators relative to the many untranslated works publishe…
DEMETR: Diagnosing Evaluation Metrics for Translation
Marzena Karpinska, Nishant Raj, Katherine Thai +3
While machine translation evaluation metrics based on string overlap (e.g., BLEU) have their limitations, their computations are transparent: the BLEU score assigned to a particula…
ezCoref: Towards Unifying Annotation Guidelines for Coreference Resolution
Ankita Gupta, Marzena Karpinska, Wenlong Zhao +5
Large-scale, high-quality corpora are critical for advancing research in coreference resolution. However, existing datasets vary in their definition of coreferences and have been c…
The Perils of Using Mechanical Turk to Evaluate Open-Ended Text Generation
Marzena Karpinska, Nader Akoury, Mohit Iyyer
Recent text generation research has increasingly focused on open-ended domains such as story and poetry generation. Because models built for such tasks are difficult to evaluate au…
NarrativeTime: Dense Temporal Annotation on a Timeline
Anna Rogers, Marzena Karpinska, Ankita Gupta +3
For the past decade, temporal annotation has been sparse: only a small portion of event pairs in a text was annotated. We present NarrativeTime, the first timeline-based annotation…