Beyond Chat and Clicks: GUI Agents for In-Situ Assistance via Live Interface Transformation

Pan Hao, Rishi Selvakumaran, Jacob Sun +1

Complex visual interfaces are powerful yet have a steep learning curve, as users must navigate feature-rich visual interfaces while reasoning about domain-specific operations. Exis…