11 papers
OpenMobile-2: Building Versatile Mobile Agents with Scalable Environments and App-Native Tools
Chenyang Yan, Yingying Zhang, Kanzhi Cheng +21
We introduce OpenMobile-2, a near-frontier mobile agent with fully open training environments and recipes. We make three key advances: (1) Diverse environments with simulated comme…
Paint What You See: Benchmarking Dexterous Visual Tool Use in Multimodal Agents
Shudong Liu, Dongyang Chen, Enci Zhang +3
Evaluation is shifting from static QA toward agentic settings where models act through external tools. We identify a critical yet underexplored capability within this space - dexte…
OpenVisTool: An Open Recipe for Synthesizing Instructive Visual Tool-Use Trajectories
Changhao Xiang, Shilin Zhang, Zheng Ma +8
Visual tool use has emerged as a fundamental capability for multimodal agents to actively acquire evidence beyond a fixed image encoding. The prevailing recipe learns this capabili…
OpenMobile: Building Open Mobile Agents with Task and Trajectory Synthesis
Kanzhi Cheng, Zehao Li, Zheng Ma +11
Mobile agents powered by vision-language models have demonstrated impressive capabilities in automating mobile tasks, with recent leading models achieving a marked performance leap…
CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era
Kanzhi Cheng, Wenpo Song, Jiaxin Fan +7
Image captioning has been a longstanding challenge in vision-language research. With the rise of LLMs, modern Vision-Language Models (VLMs) generate detailed and comprehensive imag…
MixRED: A Mix-lingual Relation Extraction Dataset
Lingxing Kong, Yougang Chu, Zheng Ma +3
Relation extraction is a critical task in the field of natural language processing with numerous real-world applications. Existing research primarily focuses on monolingual relatio…
Probing Commonsense Reasoning Capability of Text-to-Image Generative Models via Non-visual Description
Mianzhi Pan, Jianfei Li, Mingyue Yu +4
Commonsense reasoning, the ability to make logical assumptions about daily scenes, is one core intelligence of human beings. In this work, we present a novel task and dataset for e…
Bounding and Filling: A Fast and Flexible Framework for Image Captioning
Zheng Ma, Changxin Wang, Bo Huang +2
Most image captioning models following an autoregressive manner suffer from significant inference latency. Several models adopted a non-autoregressive manner to speed up the proces…
Food-500 Cap: A Fine-Grained Food Caption Benchmark for Evaluating Vision-Language Models
Zheng Ma, Mianzhi Pan, Wenhan Wu +4
Vision-language models (VLMs) have shown impressive performance in substantial downstream multi-modal tasks. However, only comparing the fine-tuned performance on downstream tasks…
ADS-Cap: A Framework for Accurate and Diverse Stylized Captioning with Unpaired Stylistic Corpora
Kanzhi Cheng, Zheng Ma, Shi Zong +3
Generating visually grounded image captions with specific linguistic styles using unpaired stylistic corpora is a challenging task, especially since we expect stylized captions wit…
Beyond Generic: Enhancing Image Captioning with Real-World Knowledge using Vision-Language Pre-Training Model
Kanzhi Cheng, Wenpo Song, Zheng Ma +3
Current captioning approaches tend to generate correct but "generic" descriptions that lack real-world knowledge, e.g., named entities and contextual information. Considering that…