3 citations · 15 across the 24 of their papers we have counts for
13 papers · 1 filter
See-Control: A Multimodal Agent Framework for Smartphone Interaction with a Robotic Arm
Haoyu Zhao, Weizhong Ding, Yuhao Yang +4
Recent advances in Multimodal Large Language Models (MLLMs) have enabled their use as intelligent agents for smartphone operation. However, existing methods depend on the Android D…
Adapting Like Humans: A Metacognitive Agent with Test-time Reasoning
Yang Li, Zhiyuan He, Yuxuan Huang +5
Recent Vision-Language Models (VLMs) exhibit strong perceptual reasoning abilities, yet they often struggle to adapt efficiently when encountering novel tasks at test time. In cont…
Hi-Agent: Hierarchical Vision-Language Agents for Mobile Device Control
Zhe Wu, Hongjin Lu, Junliang Xing +10
Building agents that autonomously operate mobile devices has attracted increasing attention. While Vision-Language Models (VLMs) show promise, most existing approaches rely on dire…
Succeed or Learn Slowly: Sample Efficient Off-Policy Reinforcement Learning for Mobile App Control
Georgios Papoudakis, Thomas Coste, Jianye Hao +2
Reinforcement learning (RL) using foundation models for policy approximations in multi-turn tasks remains challenging. We identify two main limitations related to sparse reward set…
Memento: Fine-tuning LLM Agents without Fine-tuning LLMs
Huichi Zhou, Yihang Chen, Siyuan Guo +8
In this paper, we introduce a novel learning paradigm for Adaptive Large Language Model (LLM) agents that eliminates the need for fine-tuning the underlying LLMs. Existing approach…
Uncertainty-quantified Rollout Policy Adaptation for Unlabelled Cross-domain Temporal Grounding
Jian Hu, Zixu Cheng, Shaogang Gong +4
Video Temporal Grounding (TG) aims to temporally locate video segments matching a natural language description (a query) in a long video. While Vision-Language Models (VLMs) are ef…
SpatialViz-Bench: A Cognitively-Grounded Benchmark for Diagnosing Spatial Visualization in MLLMs
Siting Wang, Minnan Pei, Luoyang Sun +6
Humans can imagine and manipulate visual images mentally, a capability known as spatial visualization. While many multi-modal benchmarks assess reasoning on visible visual informat…
Deep Research Agents: A Systematic Examination And Roadmap
Yuxuan Huang, Yihang Chen, Haozheng Zhang +10
The rapid progress of Large Language Models (LLMs) has given rise to a new category of autonomous AI systems, referred to as Deep Research (DR) agents. These agents are designed to…
Beyond Syntax: Action Semantics Learning for App Agents
Bohan Tang, Dezhao Luo, Jianheng Liu +5
The recent development of Large Language Models (LLMs) enables the rise of App agents that interpret user intent and operate smartphone Apps through actions such as clicking and sc…
AgentSwift: Efficient LLM Agent Design via Value-guided Hierarchical Search
Yu Li, Lehui Li, Zhihao Wu +5
Large language model (LLM) agents have demonstrated strong capabilities across diverse domains, yet automated agent design remains a significant challenge. Current automated agent…
ThinkBench: Dynamic Out-of-Distribution Evaluation for Robust LLM Reasoning
Shulin Huang, Linyi Yang, Yan Song +9
Evaluating large language models (LLMs) poses significant challenges, particularly due to issues of data contamination and the leakage of correct answers. To address these challeng…
AppVLM: A Lightweight Vision Language Model for Online App Control
Georgios Papoudakis, Thomas Coste, Zhihao Wu +3
The utilisation of foundation models as smartphone assistants, termed app agents, is a critical research challenge. These agents aim to execute human instructions on smartphones by…
Advancing Autonomous VLM Agents via Variational Subgoal-Conditioned Reinforcement Learning
Qingyuan Wu, Jianheng Liu, Jianye Hao +2
State-of-the-art (SOTA) reinforcement learning (RL) methods have enabled vision-language model (VLM) agents to learn from interaction with online environments without human supervi…