61 citations · 101 across the 13 of their papers we have counts for
15 papers
Impact of Expertise and Trust on AI-assisted Work
Qianou Ma, Pranav Khape, Rock Pang +4
Artificial intelligence (AI) agents promise to extend what users can accomplish beyond their existing expertise, but they also raise the risk of producing plausible-looking work th…
Better Harnesses, Smaller Models: Building 90% Cheaper Agents via Automated Harness Adaptation
Chenyang Yang, Xinran Zhao, Tongshuang Wu +1
Frontier LLM agents are automating many business tasks, but their high inference cost makes large-scale deployment unsustainable. Small language models (SLMs) offer a cheaper alter…
OdysSim: Building Foundation Models for Human Behavior Simulation
Xuhui Zhou, Weiwei Sun, Weihua Du +6
Large language models are increasingly deployed as human simulators for interactive evaluation and social simulation. Yet helpfulness-driven post-training pulls them toward a homog…
RECAP: An End-to-End Platform for Capturing, Replaying, and Analyzing AI-Assisted Programming Interactions
Keyu He, Qianou Ma, Valerie Chen +2
Understanding how developers interact with AI coding assistants requires more than chat logs or git histories in isolation; it requires reconstructing the full context: which promp…
Comparing Developer and LLM Biases in Code Evaluation
Aditya Mittal, Ryan Shar, Zichu Wu +6
As LLMs are increasingly used as judges in code applications, they should be evaluated in realistic interactive settings that capture partial context and ambiguous intent. We prese…
Not Everyone Wins with LLMs: Behavioral Patterns and Pedagogical Implications for AI Literacy in Programmatic Data Science
Qianou Ma, Kenneth Koedinger, Tongshuang Wu
LLMs promise to democratize technical work in complex domains like programmatic data analysis, but not everyone benefits equally. We study how students with varied experiences use…
"GenAI Defaults to Bias!" Gamify AI Literacy Through Reflections on Prompts
Qianou Ma, Megan Chai, Yike Tan +5
As Generative AI (GenAI) becomes widespread, it is increasingly important for the public to understand the model's behaviors and biases. However, existing AI literacy efforts miss…
What Prompts Don't Say: Understanding and Managing Underspecification in LLM Prompts
Chenyang Yang, Yike Shi, Qianou Ma +3
Prompt underspecification is a common challenge when interacting with LLMs. In this paper, we present an in-depth analysis of this problem, showing that while LLMs can often infer…
SPHERE: An Evaluation Card for Human-AI Systems
Qianou Ma, Dora Zhao, Xinran Zhao +6
In the era of Large Language Models (LLMs), establishing effective evaluation methods and standards for diverse human-AI interaction systems is increasingly challenging. To encoura…
Improving Automated Feedback Systems for Tutor Training in Low-Resource Scenarios through Data Augmentation
Chentianye Xu, Jionghao Lin, Tongshuang Wu +2
Tutoring is an effective instructional method for enhancing student learning, yet its success relies on the skill and experience of the tutors. This reliance presents challenges fo…
Orbit: A Framework for Designing and Evaluating Multi-objective Rankers
Chenyang Yang, Tesi Xiao, Michael Shavlovsky +2
Machine learning in production needs to balance multiple objectives: This is particularly evident in ranking or recommendation models, where conflicting objectives such as user eng…
What Is Wrong with My Model? Identifying Systematic Problems with Semantic Data Slicing
Chenyang Yang, Yining Hong, Grace A. Lewis +2
Machine learning models make mistakes, yet sometimes it is difficult to identify the systematic problems behind the mistakes. Practitioners engage in various activities, including…
What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use
Qianou Ma, Weirui Peng, Chenyang Yang +3
Prompting LLMs for complex tasks (e.g., building a trip advisor chatbot) needs humans to clearly articulate customized requirements (e.g., "start the response with a tl;dr"). Howev…
Beyond Testers' Biases: Guiding Model Testing with Knowledge Bases using LLMs
Chenyang Yang, Rishabh Rustogi, Rachel Brower-Sinning +3
Current model testing work has mostly focused on creating test cases. Identifying what to test is a step that is largely ignored and poorly supported. We propose Weaver, an interac…
How to Teach Programming in the AI Era? Using LLMs as a Teachable Agent for Debugging
Qianou Ma, Hua Shen, Kenneth Koedinger +1
Large Language Models (LLMs) now excel at generative skills and can create content at impeccable speeds. However, they are imperfect and still make various mistakes. In a Computer…