361 citations · 371 across the 10 of their papers we have counts for
11 papers
ProText: A benchmark dataset for measuring (mis)gendering in long-form texts
Hadas Kotek, Margit Bowler, Patrick Sonnenberg +1
We introduce ProText, a dataset for measuring gendering and misgendering in stylistically diverse long-form English texts. ProText spans three dimensions: Theme nouns (names, occup…
Apple Intelligence Foundation Language Models: Tech Report 2025
Ethan Li, Anders Boesen Lindbo Larsen, Chen Zhang +395
We introduce two multilingual, multimodal foundation language models that power Apple Intelligence features across Apple devices and services: i a 3B-parameter on-device model opti…
LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinations
Hadas Orgad, Michael Toker, Zorik Gekhman +4
Large language models (LLMs) often produce errors, including factual inaccuracies, biases, and reasoning failures, collectively referred to as "hallucinations". Recent studies have…
Apple Intelligence Foundation Language Models
Tom Gunter, Zirui Wang, Chong Wang +152
We present foundation language models developed to power Apple Intelligence features, including a ~3 billion parameter model designed to run efficiently on devices and a large serv…
Protected group bias and stereotypes in Large Language Models
Hadas Kotek, David Q. Sun, Zidi Xiu +2
As modern Large Language Models (LLMs) shatter many state-of-the-art benchmarks in a variety of domains, this paper investigates their behavior in the domains of ethics and fairnes…
DELPHI: Data for Evaluating LLMs' Performance in Handling Controversial Issues
David Q. Sun, Artem Abzaliev, Hadas Kotek +3
Controversy is a reflection of our zeitgeist, and an important aspect to any discourse. The rise of large language models (LLMs) as conversational systems has increased public reli…
Gender bias and stereotypes in Large Language Models
Hadas Kotek, Rikker Dockum, David Q. Sun
Large Language Models (LLMs) have made substantial progress in the past several months, shattering state-of-the-art benchmarks in many domains. This paper investigates LLMs' behavi…
Feedback Effect in User Interaction with Intelligent Assistants: Delayed Engagement, Adaption and Drop-out
Zidi Xiu, Kai-Chen Cheng, David Q. Sun +7
With the growing popularity of intelligent assistants (IAs), evaluating IA quality becomes an increasingly active field of research. This paper identifies and quantifies the feedba…
MMIU: Dataset for Visual Intent Understanding in Multimodal Assistants
Alkesh Patel, Joel Ruben Antony Moniz, Roman Nguyen +3
In multimodal assistant, where vision is also one of the input modalities, the identification of user intent becomes a challenging task as visual input can influence the outcome. C…
Improving Human-Labeled Data through Dynamic Automatic Conflict Resolution
David Q. Sun, Hadas Kotek, Christopher Klein +3
This paper develops and implements a scalable methodology for (a) estimating the noisiness of labels produced by a typical crowdsourcing semantic annotation task, and (b) reducing…
Generating Natural Questions from Images for Multimodal Assistants
Alkesh Patel, Akanksha Bindal, Hadas Kotek +2
Generating natural, diverse, and meaningful questions from images is an essential task for multimodal assistants as it confirms whether they have understood the object and scene in…