DashAct: A Progressive Diagnostic Benchmark for GUI Agents in Interactive Dashboard Analysis

Abstract

Interactive dashboards require users to reveal and connect evidence across stateful interactions. Although graphical user interface (GUI) agents could automate this process, existing dashboard benchmarks primarily report final answers or task success. They provide limited insight into whether failures arise from maintaining the analytical process, selecting actions, or grounding visual targets. We introduce DashAct, a progressive diagnostic benchmark that evaluates the same annotated dashboard interactions under systematically varied contextual and spatial support. DashAct contains 348 human-verified interaction trajectories with milestone dependencies and hierarchical target annotations. Its three stages evaluate end-to-end execution, verified-context action prediction, and target-conditioned grounding under four visual conditions. Aligned action and spatial annotations connect outcomes across these settings, providing diagnostic evidence for interpreting online failures. Experiments reveal persistent difficulties in action prediction and fine-grained grounding, alongside model-dependent responses to contextual and spatial support. A Qwen-based adaptation study further improves target localization and online Milestone Coverage, illustrating how DashAct can guide practical agent improvements.