MAESTRO: Adapting GUIs and Guiding Navigation with User Preferences in Conversational Agents with GUIs

Abstract

Modern task-oriented chatbots present GUI elements alongside natural-language dialogue, yet the agent's role has largely been limited to interpreting natural-language input as GUI actions and following a linear workflow. In preference-driven, multi-step tasks such as booking a flight or reserving a restaurant, earlier choices constrain later options and may force users to restart from scratch. User preferences serve as the key criteria for these decisions, yet existing agents do not systematically leverage them. We present MAESTRO, which extends the agent's role from execution to decision support. MAESTRO maintains a shared preference memory that extracts hard and soft preferences from natural-language utterances and provides two mechanisms. Preference-Grounded GUI Adaptation applies in-place operators (augment, sort, filter, and highlight) to the existing GUI according to preference strength, supporting comparison among options. Preference-Guided Workflow Navigation detects conflicts between preferences and available options, proposes backtracking, and records failed paths to avoid revisiting dead ends. Through a controlled experiment (N=33), we demonstrated that MAESTRO improved decision quality in movie ticketing: final bookings left fewer hard preferences unmet, and users made fewer selections that violated their stated preferences during the process than in the baseline condition, although task success rate and completion time did not differ significantly. In addition, we showed that using MAESTRO in voice mode can increase users' active engagement as well as their mental burden, revealing the nuanced tension in agentic interaction design for conversational agents with a GUI.

19 pages, 6 figures, 2 tables. Published at UIST '26. This is the camera-ready version, posted under CC BY 4.0