16 papers
Guilt by Association: Semantic Leakage Drives Associative Gender Bias
Hazel Chen, Peter West, Hila Gonen
Semantic leakage occurs when a concept in the prompt steers model generation about an entity, even though the concept is irrelevant to it (e.g., a person who likes yellow is said t…
DeepRewind: Predicting and Repairing Premature Commitments in Deep Research Agents
Amirhossein Abaskohi, Amirhossein Dabiriaghdam, Lele Wang +2
Deep-research agents conduct long-horizon investigations through iterative search, evidence evaluation, belief revision, and synthesis. However, they may commit to claims before su…
Is Convergence Inevitable? Tracing Output Homogeneity Back to Base Models
Alexandrine Fortier, Hazel Chen, Peter West
The lack of diversity in LM content is widely attributed to the alignment process, but how and where exactly in the pipeline this collapse begins is unknown. We argue that output h…
UnpredictaBench: A Benchmark for Evaluating Distributional Randomness in LLMs
Amirhossein Abaskohi, Amirhossein Dabiriaghdam, Liang Luo +4
We introduce UnpredictaBench, an evaluation that tests the ability of large language models (LLMs) to capture true underlying distributions. As LLMs are increasingly used as substi…
SeKV: Resolution-Adaptive KV Cache with Hierarchical Semantic Memory for Long-Context LLM Inference
Amirhossein Abaskohi, Giuseppe Carenini, Peter West +1
Large language models increasingly operate over long contexts, where the KV cache becomes a dominant memory bottleneck: its size grows linearly with sequence length and must be ret…
Towards Physical Intuitions for Alignment Dynamics: A Case Study With Randomness Crystallization
Kunal Samanta, Ari Holtzman, Peter West
The alignment of language models is typically studied through the lens of capability benchmarks, but the dynamics of how models change during post-training remain poorly understood…
Resolution Thresholds in VLM Detection of Harmful ASCII Art Across Construction Modes and Languages
Yikai Hua, Peter West
Large Vision-Language Models (VLMs) are increasingly deployed as content moderation tools, yet they remain vulnerable to jailbreak attacks in which harmful text is visually encoded…
MCompassRAG: Topic Metadata as a Semantic Compass for Paragraph-Level Retrieval
Amirhossein Abaskohi, Raymond Li, Gaetano Cimino +3
Retrieval-augmented generation (RAG) systems depend critically on how documents are chunked and searched. Fine-grained chunks can improve retrieval precision but expand the search…
SproutRAG: Attention-Guided Tree Search with Progressive Embeddings for Long-Document RAG
Amirhossein Abaskohi, Issam H. Laradji, Peter West +1
Retrieval-augmented generation (RAG) systems must balance retrieval granularity with contextual coherence, a challenge that existing methods address through LLM-guided chunking, si…
ReVision: Scaling Computer-Use Agents via Temporal Visual Redundancy Reduction
Amirhossein Abaskohi, Yuhang He, Peter West +3
Computer-use agents (CUAs) rely on visual observations of graphical user interfaces, where each screenshot is encoded into a large number of visual tokens. As interaction trajector…
Can You Keep a Secret? Involuntary Information Leakage in Language Model Writing
Ari Holtzman, Peter West
Language models are deployed in settings that require compartmentalization: system prompts should not be disclosed, chain-of-thought reasoning is hidden from users, and sensitive d…
Moral Mazes in the Era of LLMs
Dang Nguyen, Harvey Yiyun Fu, Peter West +2
Navigating complex social situations is an integral part of corporate life, ranging from giving critical feedback without hurting morale to rejecting requests without alienating te…
Where Do Backdoors Live? A Component-Level Analysis of Backdoor Propagation in Speech Language Models
Alexandrine Fortier, Thomas Thebaud, Jesús Villalba +3
Speech language models (SLMs) are systems of systems: independent components that unite to achieve a common goal. Despite their heterogeneous nature, SLMs are often studied end-to-…
AbsenceBench: Language Models Can't Tell What's Missing
Harvey Yiyun Fu, Aryan Shrivastava, Jared Moore +3
Large language models (LLMs) are increasingly capable of processing long inputs and locating specific information within them, as evidenced by their performance on the Needle in a…
Benchmarks as Microscopes: A Call for Model Metrology
Michael Saxon, Ari Holtzman, Peter West +2
Modern language models (LMs) pose a new challenge in capability assessment. Static benchmarks inevitably saturate without providing confidence in the deployment tolerances of LM-ba…
Predicting vs. Acting: A Trade-off Between World Modeling & Agent Modeling
Margaret Li, Weijia Shi, Artidoro Pagnoni +2
RLHF-aligned LMs have shown unprecedented ability on both benchmarks and long-form text generation, yet they struggle with one foundational task: next-token prediction. As RLHF mod…