16 citations · 60 across the 23 of their papers we have counts for
24 papers
"Just Like This": Manner Deixis at the Graphics Interface
Hamza El Alaoui, Jeffrey P. Bigham, Jun Rekimoto
People communicate how things should move by combining words with demonstrations: "open it like this." We present an interaction technique that brings this expressive resource to c…
When Disability Disclosure Travels: Memory, Privacy, and Contextual Integrity in Conversational AI
Atieh Taheri, Mahya Tazike, Patrick Carrington +1
Conversational AI assistants remember what people tell them, and for disabled people, that often includes disability. We interviewed 12 adults with disabilities in the United State…
Quantifying the Statistical Effect of Rubric Modifications on Human-Autorater Agreement
Jessica Huynh, Alfredo Gomez, Athiya Deviyani +3
Autoraters, also referred to as LLM-as-judges, are increasingly used for evaluation and automated content moderation. However, there is limited statistical analysis of how modifica…
Efficient Personalization of Generative User Interfaces
Yi-Hao Peng, Samarth Das, Jeffrey P. Bigham +1
Generative user interfaces (GenUIs) create new opportunities to adapt interfaces to individual users on demand. Yet personalization is difficult because it is not possible to provi…
"I followed what felt right, not what I was told": Autonomy, Coaching, and Recognizing Bias Through AI-Mediated Dialogue
Atieh Taheri, Hamza El Alaoui, Patrick Carrington +1
Ableist microaggressions remain pervasive in everyday interactions, yet interventions to help people recognize them are limited. We present an experiment testing how AI-mediated di…
Modeling Distinct Human Interaction in Web Agents
Faria Huq, Zora Zhiruo Wang, Zhanqiu Guo +6
Despite rapid progress in autonomous web agents, human involvement remains essential for shaping preferences and correcting agent behavior as tasks unfold. However, current agentic…
DesignPref: Capturing Personal Preferences in Visual Design Generation
Yi-Hao Peng, Jeffrey P. Bigham, Jason Wu
Generative models, such as large language models and text-to-image diffusion models, are increasingly used to create visual designs like user interfaces (UIs) and presentation slid…
VLSU: Mapping the Limits of Joint Multimodal Understanding for AI Safety
Shruti Palaskar, Leon Gatys, Mona Abdelrahman +9
Safety evaluation of multimodal foundation models often treats vision and language inputs separately, missing risks from joint interpretation where benign content becomes harmful i…
Morae: Proactively Pausing UI Agents for User Choices
Yi-Hao Peng, Dingzeyu Li, Jeffrey P. Bigham +1
User interface (UI) agents promise to make inaccessible or complex UIs easier to access for blind and low-vision (BLV) users. However, current UI agents typically perform tasks end…
Apple Intelligence Foundation Language Models: Tech Report 2025
Ethan Li, Anders Boesen Lindbo Larsen, Chen Zhang +395
We introduce two multilingual, multimodal foundation language models that power Apple Intelligence features across Apple devices and services: i a 3B-parameter on-device model opti…
StepWrite: Adaptive Planning for Speech-Driven Text Generation
Hamza El Alaoui, Atieh Taheri, Yi-Hao Peng +1
People frequently use speech-to-text systems to compose short texts with voice. However, current voice-based interfaces struggle to support composing more detailed, contextually co…
CodeA11y: Making AI Coding Assistants Useful for Accessible Web Development
Peya Mowar, Yi-Hao Peng, Jason Wu +2
A persistent challenge in accessible computing is ensuring developers produce web UI code that supports assistive technologies. Despite numerous specialized accessibility tools, no…
CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web Navigation
Faria Huq, Zora Zhiruo Wang, Frank F. Xu +4
While much work on web agents emphasizes the promise of autonomously performing tasks on behalf of users, in reality, agents often fall short on complex tasks in real-world context…
DreamStruct: Understanding Slides and User Interfaces via Synthetic Data Generation
Yi-Hao Peng, Faria Huq, Yue Jiang +4
Enabling machines to understand structured visuals like slides and user interfaces is essential for making them accessible to people with disabilities. However, achieving such unde…
Policy Maps: Tools for Guiding the Unbounded Space of LLM Behaviors
Michelle S. Lam, Fred Hohman, Dominik Moritz +3
AI policy sets boundaries on acceptable behavior for AI models, but this is challenging in the context of large language models (LLMs): how do you ensure coverage over a vast behav…
NoTeeline: Supporting Real-Time, Personalized Notetaking with LLM-Enhanced Micronotes
Faria Huq, Abdus Samee, David Chuan-en Lin +2
Taking notes quickly while effectively capturing key information can be challenging, especially when watching videos that present simultaneous visual and auditory streams. Manually…
Exploring the Role of Social Support when Integrating Generative AI into Small Business Workflows
Quentin Romero Lauro, Jeffrey P. Bigham, Yasmine Kotturi
Small business owners stand to benefit from generative AI technologies due to limited resources, yet they must navigate increasing legal and ethical risks. In this paper, we interv…
Apple Intelligence Foundation Language Models
Tom Gunter, Zirui Wang, Chong Wang +152
We present foundation language models developed to power Apple Intelligence features, including a ~3 billion parameter model designed to run efficiently on devices and a large serv…
UICoder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedback
Jason Wu, Eldon Schoop, Alan Leung +3
Large language models (LLMs) struggle to consistently generate UI code that compiles and produces visually relevant designs. Existing approaches to improve generation rely on expen…
Position: Towards Bidirectional Human-AI Alignment
Hua Shen, Tiffany Knearem, Reshmi Ghosh +21
Recent advances in general-purpose AI underscore the urgent need to align AI systems with human goals and values. Yet, the lack of a clear, shared understanding of what constitutes…
"This really lets us see the entire world:" Designing a conversational telepresence robot for homebound older adults
Yaxin Hu, Laura Stegner, Yasmine Kotturi +6
In this paper, we explore the design and use of conversational telepresence robots to help homebound older adults interact with the external world. An initial needfinding study (N=…
UIClip: A Data-driven Model for Assessing User Interface Design
Jason Wu, Yi-Hao Peng, Amanda Li +3
User interface (UI) design is a difficult yet important task for ensuring the usability, accessibility, and aesthetic qualities of applications. In our paper, we develop a machine-…
Talaria: Interactively Optimizing Machine Learning Models for Efficient Inference
Fred Hohman, Chaoqun Wang, Jinmook Lee +7
On-device machine learning (ML) moves computation from the cloud to personal devices, protecting user privacy and enabling intelligent user experiences. However, fitting models on…
Towards Automated Accessibility Report Generation for Mobile Apps
Amanda Swearngin, Jason Wu, Xiaoyi Zhang +8
Many apps have basic accessibility issues, like missing labels or low contrast. Automated tools can help app developers catch basic issues, but can be laborious or require writing…