53 citations · 98 across the 21 of their papers we have counts for
4 papers · 1 filter
Ego2Web: A Web Agent Benchmark Grounded in Egocentric Videos
Shoubin Yu, Lei Shu, Antoine Yang +6
Multimodal AI agents are increasingly automating complex real-world workflows that involve online web execution. However, current web-agent benchmarks suffer from a critical limita…
UISim: An Interactive Image-Based UI Simulator for Dynamic Mobile Environments
Jiannan Xiang, Yun Zhu, Lei Shu +6
Developing and testing user interfaces (UIs) and training AI agents to interact with them are challenging due to the dynamic and diverse nature of real-world mobile environments. E…
ScreenAI: A Vision-Language Model for UI and Infographics Understanding
Gilles Baechler, Srinivas Sunkara, Maria Wang +7
Screen user interfaces (UIs) and infographics, sharing similar visual language and design principles, play important roles in human communication and human-machine interaction. We…
UIBert: Learning Generic Multimodal Representations for UI Understanding
Chongyang Bai, Xiaoxue Zang, Ying Xu +4
To improve the accessibility of smart devices and to simplify their usage, building models which understand user interfaces (UIs) and assist users to complete their tasks is critic…