UI-JEPA: Towards Active Perception of User Intent through Onscreen User Activity

Yicheng Fu, Raviteja Anantha, Prabal Vashisht +2

Generating user intent from a sequence of user interface (UI) actions is a core challenge in comprehensive UI understanding. Recent advancements in multimodal large language models…