MUG: Interactive Multimodal Grounding on User Interfaces

Tao Li, Gang Li, Jingjie Zheng +2

We present MUG, a novel interactive task for multimodal grounding where a user and an agent work collaboratively on an interface screen. Prior works modeled multimodal UI grounding…