5 papers
Syll: Open-Source Personal Automation with Cross-Surface Execution
Bo Zhang, Borui Zhang, Chenghao Jiang +5
Personal AI agents must increasingly operate across APIs, shells, web surfaces, and desktop GUIs, yet many systems remain tuned to a single interface and offer limited support for…
AdaZoom-GUI: Adaptive Zoom-based GUI Grounding with Instruction Refinement
Siqi Pei, Liang Tang, Tiaonan Duan +9
GUI grounding is a critical capability for vision-language models (VLMs) that enables automated interaction with graphical user interfaces by locating target elements from natural…
Point and Go: Intuitive Reference Frame Reallocation in Mode Switching for Assistive Robotics
A. Wang, C. Jiang, M. Przystupa +2
Operating high degree of freedom robots can be difficult for users of wheelchair mounted robotic manipulators. Mode switching in Cartesian space has several drawbacks such as unint…
Robot Manipulation in Salient Vision through Referring Image Segmentation and Geometric Constraints
Chen Jiang, Allie Luo, Martin Jagersand
In this paper, we perform robot manipulation activities in real-world environments with language contexts by integrating a compact referring image segmentation model into the robot…
Interpreting Behaviors and Geometric Constraints as Knowledge Graphs for Robot Manipulation Control
Chen Jiang, Allie Wang, Martin Jagersand
In this paper, we investigate the feasibility of using knowledge graphs to interpret actions and behaviors for robot manipulation control. Equipped with an uncalibrated visual serv…