13 papers
JarvisGUI: Towards Cross-Device GUI Agents with Dynamic Task Composition
Zixiang Chen, Yuheng Lu, Zihao Cheng +8
Real-world GUI usage frequently involves workflows that span multiple devices and platforms, requiring the transfer of intermediate results, maintenance of shared state, and coordi…
Towards Generalizable Visually Grounded Exploration of Household Devices
Linhao Zheng, Zeming Liu, Wangke Chen +4
Recent advancements in Vision-Language Models (VLMs) have demonstrated impressive capabilities in static visual recognition and high-level semantic reasoning. However, current embo…
Behavior2Trip: Towards Personalized Travel Planning via User Behavior Trajectory
Zihao Cheng, Yingyu Shan, Hongru Wang +6
Travel planning agents assist users in generating personalized travel plans by modeling their individual preferences. Existing agents either rely on explicit user instructions or e…
Learning from Own Solutions: Self-Conditioned Credit Assignment for Reinforcement Learning with Verifiable Rewards
Yingyu Shan, Yuhang Guo, Zihao Cheng +7
Reinforcement learning with verifiable rewards (RLVR) has driven substantial progress in training LLMs for reasoning tasks, but representative methods such as GRPO assign uniform c…
PEC-Home: Interpretation of Progressively Elliptical Commands in Smart Homes
Yingyu Shan, Zeming Liu, Silin Li +4
Recent advancements in Large Language Models (LLMs) have empowered home assistants with natural language interaction capabilities. However, current assistants overlook the progress…
Benchmarking Living-Screen-Native GUI Agents on Short-Video Platforms
Jiashu Yao, Heyan Huang, Daiqing Wu +5
GUI agents today assume a static screen, where the world is frozen between two actions. However, real interfaces such as short-video applications violate this assumption, as their…
Terminal-World: Scaling Terminal-Agent Environments via Agent Skills
Zihao Cheng, Hongru Wang, Zeming Liu +6
Terminal agents extend Large Language Models with the ability to execute tasks directly in command-line environments, but their progress is bottlenecked by the scarcity of high-qua…
DocOS: Towards Proactive Document-Guided Actions in GUI Agents
Jingjing Liu, Ziye Huang, Zihao Cheng +6
While Graphical User Interface (GUI) agents have shown promising performance in automated device interaction, they primarily depend on static parametric knowledge from pre-training…
Utilizing and Calibrating Hindsight Process Rewards via Reinforcement with Mutual Information Self-Evaluation
Jiashu Yao, Heyan Huang, Zeming Liu +1
To overcome the sparse reward challenge in reinforcement learning (RL) for agents based on large language models (LLMs), we propose Mutual Information Self-Evaluation (MISE), an RL…
MemEvolve: Towards Self-Evolving Agents via Co-Evolutionary Capability Expansion and Experience Distillation
Zihao Cheng, Zeming Liu, Yingyu Shan +7
While large language model--powered agents can self-evolve by accumulating experience or by dynamically creating new assets (i.e., tools or expert agents), existing frameworks typi…
Policy Split: Incentivizing Dual-Mode Exploration in LLM Reinforcement with Dual-Mode Entropy Regularization
Jiashu Yao, Heyan Huang, Daiqing Wu +2
To encourage diverse exploration in reinforcement learning (RL) for large language models (LLMs) without compromising accuracy, we propose Policy Split, a novel paradigm that bifur…
HomeSafeBench: Benchmarking Embodied Vision-Language Models in Free-Exploration Home Safety Inspection
Jiashu Yao, Haoyu Wen, Siyuan Gao +3
Safety hazards in the home are a leading cause of preventable domestic injuries, motivating an automated inspector that actively explores a home and reports hazards before they cau…
ReFF: Reinforcing Format Faithfulness in Language Models across Varied Tasks
Jiashu Yao, Heyan Huang, Zeming Liu +4
Following formatting instructions to generate well-structured content is a fundamental yet often unmet capability for large language models (LLMs). To study this capability, which…