21 papers
Adaptive-Group Process Rewards for GUI Agents
Zirui Shang, Yang Liu, Xin Shu +3
Training graphical user interface (GUI) agents with reinforcement learning suffers from sparse outcome rewards, leading to inefficient learning. Process rewards offer a promising d…
Selective Knowledge Control for Continual GUI Agent Learning over Application Streams
Zirui Shang, Xin Shu, Yang Liu +3
Continual learning is a crucial capability for Graphical User Interface (GUI) agents to adapt to evolving applications while retaining knowledge acquired from previous applications…
Beyond Sequential Interaction: Benchmarking Parallel Execution and Coordination for GUI Agents
Zedong Yu, Qianxing Li, Zhi Gao +8
Graphical user interface (GUI) agents are systems powered by large multimodal models (LMMs). They perceive screen state and execute user instructions through GUI actions such as cl…
Taming I2V models for Image HOI Editing: A Cognitive Benchmark and Agentic Self-Correcting Framework
Jiayi Gao, Qingchao Chen, Yuxin Peng +1
Current image editing methods excel at static attributes but fail at complex Human-Object Interactions (HOI), a critical challenge unaddressed by existing benchmarks that conflate…
Text as Partial Constraint: Core-Residual Alignment for Robust Vision-Language Learning
Chengzhen Yu, Canran Xiao, Siyuan Ma +1
Vision-language alignment powers open-vocabulary recognition, retrieval, and LVLM grounding, yet natural captions are often underspecified, making similarity brittle and overly con…
In-Context Model Predictive Generation: Open-Vocabulary Motion Synthesis from Language Models to Physics
Xiaomeng Fu, Junfan Lin, Yang Liu +4
Synthesizing human motion from textual descriptions is essential for immersive digital applications, yet existing methods face a persistent trade-off between semantic fidelity and…
Learning with a Single Rollout via Pass@k Policy Optimization
Fengdi Che, Yang Liu, Lei Yu +4
Group Relative Policy Optimization (GRPO) typically requires multiple rollouts per prompt to estimate relative credit, incurring substantial sampling cost and exacerbating training…
EG-VQA: Benchmarking Verifiable Video Question Answering with Grounded Temporal Evidence
Linpeng Huang, Weixing Chen, Zexin Chen +2
Recent advances in Video Large Language Models (Video-LLMs) have yielded promising performance on Video Question Answering (VideoQA). Nevertheless, existing benchmarks are predomin…
Attention-Spectrum Regularization for Replay-Free Continual Multimodal LLMs
Chuangxin Zhao, Canran Xiao, Siyuan Ma +5
Multimodal large language models (MLLMs) are increasingly required to adapt to non-stationary streams of visual domains, question types, and user instructions, yet continual fine-t…
ARP: Enhancing Quantized Skill Abstractions via Visual Alignment and Iterative Refinement for Robotic Manipulation
Yuntian Wang, Zesheng Jia, Yuhui Duan +5
Learning visuomotor policies for long-horizon manipulation remains a fundamental challenge. Recent skill-based imitation learning methods based on discrete quantization have shown…
ATLAS: Agentic Taxonomy of Large-Scale Software Ecosystems
Junyi Lu, Mengyao Lyu, Jiahui Wu +6
The open-source ecosystem on GitHub lacks a systematic hierarchical taxonomy of software repositories. GitHub Topics, the dominant organizational mechanism, is flat, inconsistent,…
RoVLA: Multi-Consistency Constraints for Robust Vision-Language-Action Models
Jingzhou Luo, Yifan Wen, Yongjie Bai +3
Vision-Language-Action (VLA) models have shown strong performance on embodied manipulation, yet they remain brittle under visual observation changes, paraphrased language instructi…
Benchmarking and Improving GUI Agents in High-Dynamic Environments
Enqi Liu, Liyuan Pan, Zhi Gao +5
Recent advancements in Graphical User Interface (GUI) agents have predominantly focused on training paradigms like supervised fine-tuning (SFT) and reinforcement learning (RL). How…
CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning
Yang Liu, Toan Nguyen, Flora D. Salim
Catastrophic forgetting remains a major obstacle to continual learning in large language models (LLMs) and vision--language models (VLMs). Although Mixture-of-Experts (MoE) archite…
Fre-Res: Frequency-Residual Video Token Compression for Efficient Video MLLMs
Yigui Feng, Qinglin Wang, Yang Liu +1
Video MLLMs face a persistent tension between spatial fidelity and temporal coverage: preserving fine-grained visual details requires many spatial tokens, while capturing short-liv…
OmniVTG: A Large-Scale Dataset and Training Paradigm for Open-World Video Temporal Grounding
Minghang Zheng, Zihao Yin, Yi Yang +2
Video Temporal Grounding (VTG), the task of localizing video segments from text queries, struggles in open-world settings due to limited dataset scale and semantic diversity, causi…
Evaluating Time Awareness and Cross-modal Active Perception of Large Models via 4D Escape Room Task
Yurui Dong, Ziyue Wang, Shuyun Lu +5
Multimodal Large Language Models (MLLMs) have recently made rapid progress toward unified Omni models that integrate vision, language, and audio. However, existing environments lar…
AGCD: Agent-Guided Cross-Modal Decoding for Weather Forecasting
Jing Wu, Yang Liu, Lin Zhang +13
Accurate weather forecasting is more than grid-wise regression: it must preserve coherent synoptic structures and physical consistency of meteorological fields, especially under au…
Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curation
Pengxiang Li, Zechen Hu, Zirui Shang +15
Vision-language model (VLM) based GUI agents show promise for automating complex desktop and mobile tasks, but face significant challenges in applying reinforcement learning (RL):…
CircuitProbe: Tracing Visual Temporal Evidence Flow in Video Language Models
Yiming Zhang, Zhuokai Zhao, Chengzhang Yu +8
Autoregressive large vision--language models (LVLMs) interface video and language by projecting video features into the LLM's embedding space as continuous visual token embeddings.…
Adaptive Chain-of-Focus Reasoning via Dynamic Visual Search and Zooming for Efficient VLMs
Xintong Zhang, Zhi Gao, Bofei Zhang +8
Vision language models (VLMs) have achieved impressive performance across a variety of computer vision tasks. However, the multimodal reasoning capability has not been fully explor…