Publications (26)
The Prompt Artists
Minsuk Chang, Stefania Druga, Alex Fiannaca +4
This paper examines the art practices, artwork, and motivations of prolific users of the latest generation of text-to-image models. Through interviews, observations, and a user sur…
TensorFlow.js: Machine Learning for the Web and Beyond
Daniel Smilkov, Nikhil Thorat, Yannick Assogba +17
TensorFlow.js is a library for building and executing machine learning algorithms in JavaScript. TensorFlow.js models run in a web browser and in the Node.js environment. The libra…
Guided Integrated Gradients: An Adaptive Path Method for Removing Noise
Andrei Kapishnikov, Subhashini Venugopalan, Besim Avci +3
Integrated Gradients (IG) is a commonly used feature attribution method for deep neural networks. While IG has many desirable properties, the method often produces spurious/noisy p…
PromptInfuser: How Tightly Coupling AI and UI Design Impacts Designers' Workflows
Savvas Petridis, Michael Terry, Carrie J. Cai
Prototyping AI applications is notoriously difficult. While large language model (LLM) prompting has dramatically lowered the barriers to AI prototyping, designers are still protot…
IMACS: Image Model Attribution Comparison Summaries
Eldon Schoop, Ben Wedin, Andrei Kapishnikov +2
Developing a suitable Deep Neural Network (DNN) often requires significant iteration, where different model versions are evaluated and compared. While metrics such as accuracy are…
ConstitutionMaker: Interactively Critiquing Large Language Models by Converting Feedback into Principles
Savvas Petridis, Ben Wedin, James Wexler +5
Large language model (LLM) prompting is a promising new approach for users to create and customize their own chatbots. However, current methods for steering a chatbot's outputs, su…
Gensors: Authoring Personalized Visual Sensors with Multimodal Foundation Models and Reasoning
Michael Xieyang Liu, Savvas Petridis, Vivian Tsai +4
Multimodal large language models (MLLMs), with their expansive world knowledge and reasoning capabilities, present a unique opportunity for end-users to create personalized AI sens…
In Situ AI Prototyping: Infusing Multimodal Prompts into Mobile Settings with MobileMaker
Savvas Petridis, Michael Xieyang Liu, Alexander J. Fiannaca +3
Recent advances in multimodal large language models (LLMs) have made it easier to rapidly prototype AI-powered features, especially for mobile use cases. However, gathering early,…
LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large Language Models
Minsuk Kahng, Ian Tenney, Mahima Pushkarna +7
Automatic side-by-side evaluation has emerged as a promising approach to evaluating the quality of responses from large language models (LLMs). However, analyzing the results from…
Compass vs Railway Tracks: Unpacking User Mental Models for Communicating Long-Horizon Work to Humans vs. AI
Savvas Petridis, Michael Xieyang Liu, Alexander J. Fiannaca +2
As AI systems grow increasingly capable of operating for hours or days at a time, users' prompts are transforming into elaborate specifications for the AI to autonomously work on.…
AutoMOS: Learning a non-intrusive assessor of naturalness-of-speech
Brian Patton, Yannis Agiomyrgiannakis, Michael Terry +3
Developers of text-to-speech synthesizers (TTS) often make use of human raters to assess the quality of synthesized speech. We demonstrate that we can model human raters' mean opin…
The Design Space of Generative Models
Meredith Ringel Morris, Carrie J. Cai, Jess Holbrook +2
Card et al.'s classic paper "The Design Space of Input Devices" established the value of design spaces as a tool for HCI analysis and invention. We posit that developing design spa…
XRAI: Better Attributions Through Regions
Andrei Kapishnikov, Tolga Bolukbasi, Fernanda Viégas +1
Saliency methods can aid understanding of deep neural networks. Recent years have witnessed many improvements to saliency methods, as well as new ways for evaluating them. In this…
Beyond Code Generation: LLM-supported Exploration of the Program Design Space
J. D. Zamfirescu-Pereira, Eunice Jun, Michael Terry +2
In this work, we explore explicit Large Language Model (LLM)-powered support for the iterative design of computer programs. Program design, like other design activity, is character…
AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts
Tongshuang Wu, Michael Terry, Carrie J. Cai
Although large language models (LLMs) have demonstrated impressive potential on simple tasks, their breadth of scope, lack of transparency, and insufficient controllability can mak…
PromptChainer: Chaining Large Language Model Prompts through Visual Programming
Tongshuang Wu, Ellen Jiang, Aaron Donsbach +4
While LLMs can effectively help prototype single ML functionalities, many real-world applications involve complex tasks that cannot be easily handled via a single run of an LLM. Re…
"We Need Structured Output": Towards User-centered Constraints on Large Language Model Output
Michael Xieyang Liu, Frederick Liu, Alexander J. Fiannaca +4
Large language models can produce creative and diverse responses. However, to integrate them into current developer workflows, it is essential to constrain their outputs to follow…
Designing for Human-Agent Alignment: Understanding what humans want from their agents
Nitesh Goyal, Minsuk Chang, Michael Terry
Our ability to build autonomous agents that leverage Generative AI continues to increase by the day. As builders and users of such agents it is unclear what parameters we need to a…
Human-Centered Tools for Coping with Imperfect Algorithms during Medical Decision-Making
Carrie J. Cai, Emily Reif, Narayan Hegde +8
Machine learning (ML) is increasingly being used in image retrieval systems for medical decision making. One application of ML is to retrieve visually similar medical images from p…
Position: Towards Bidirectional Human-AI Alignment
Hua Shen, Tiffany Knearem, Reshmi Ghosh +21
Recent advances in general-purpose AI underscore the urgent need to align AI systems with human goals and values. Yet, the lack of a clear, shared understanding of what constitutes…
Similar Image Search for Histopathology: SMILY
Narayan Hegde, Jason D. Hipp, Yun Liu +11
The increasing availability of large institutional and public histopathology image datasets is enabling the searching of these datasets for diagnosis, research, and education. Thou…
The Evolution of LLM Adoption in Industry Data Curation Practices
Crystal Qian, Michael Xieyang Liu, Emily Reif +7
As large language models (LLMs) grow increasingly adept at processing unstructured text data, they offer new opportunities to enhance data curation workflows. This paper explores t…
Farsight: Fostering Responsible AI Awareness During AI Application Prototyping
Zijie J. Wang, Chinmay Kulkarni, Lauren Wilcox +2
Prompt-based interfaces for Large Language Models (LLMs) have made prototyping and building AI-powered applications easier than ever before. However, identifying potential harms th…
Interactive AI Alignment: Specification, Process, and Evaluation Alignment
Michael Terry, Chinmay Kulkarni, Martin Wattenberg +2
Modern AI enables a high-level, declarative form of interaction: Users describe the intended outcome they wish an AI to produce, but do not actually create the outcome themselves.…
A Word is Worth a Thousand Pictures: Prompts as AI Design Material
Chinmay Kulkarni, Stefania Druga, Minsuk Chang +3
Recent advances in Machine-Learning have led to the development of models that generate images based on a text description.Such large prompt-based text to image models (TTIs), trai…
Program Synthesis with Large Language Models
Jacob Austin, Augustus Odena, Maxwell Nye +8
This paper explores the limits of the current generation of large language models for program synthesis in general purpose programming languages. We evaluate a collection of such m…