23 papers
AIR: Adaptive Interleaved Reasoning with Code in MLLMs
Cong Han, Xiaohan Lan, Haibo Qiu +1
Following the paradigm shift initiated by OpenAI o3, interleaved reasoning with code to enhance multimodal large language models (MLLMs) has become a pivotal research frontier. The…
Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence
Xuanle Zhao, Qiushi Sun, Jingyu Xiao +16
While Large Language Models (LLMs) have substantially advanced text-to-code generation, many real programming tasks specify intent through visual artifacts such as screenshots, cha…
Seirênes: Adversarial Self-Play with Evolving Distractions for LLM Reasoning
Chi Zhang, Haibo Qiu, Qiming Zhang +3
We present Seirênes, a self-play RL framework that transforms contextual interference from a failure mode of LLM reasoning into an internal training signal for co-evolving more res…
SkillFlow:Benchmarking Lifelong Skill Discovery and Evolution for Autonomous Agents
Ziao Zhang, Kou Shi, Shiting Huang +13
As the capability frontier of autonomous agents continues to expand, they are increasingly able to complete specialized tasks through plug-and-play external skills. Yet current ben…
Omni-I2C: A Holistic Benchmark for High-Fidelity Image-to-Code Generation
Jiawei Zhou, Chi Zhang, Xiang Feng +6
We present Omni-I2C, a comprehensive benchmark designed to evaluate the capability of Large Multimodal Models (LMMs) in converting complex, structured digital graphics into executa…
TreeCUA: Efficiently Scaling GUI Automation with Tree-Structured Verifiable Evolution
Deyang Jiang, Jing Huang, Xuanle Zhao +6
Effectively scaling GUI automation is essential for computer-use agents (CUAs); however, existing work primarily focuses on scaling GUI grounding rather than the more crucial GUI p…
Length-Unbiased Sequence Policy Optimization: Revealing and Controlling Response Length Variation in RLVR
Fanfan Liu, Youyang Yin, Peng Shi +3
Recent applications of Reinforcement Learning with Verifiable Rewards (RLVR) to Large Language Models (LLMs) and Vision-Language Models (VLMs) have demonstrated significant success…
Flexible Entropy Control in RLVR with a Gradient-Preserving Perspective
Kun Chen, Peng Shi, Fanfan Liu +4
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a critical method for enhancing the reasoning capabilities of Large Language Models (LLMs). However, continuous…
MobileDreamer: Generative Sketch World Model for GUI Agent
Yilin Cao, Yufeng Zhong, Zhixiong Zeng +6
Mobile GUI agents have shown strong potential in real-world automation and practical applications. However, most existing agents remain reactive, making decisions mainly from curre…
Learning When to Look: A Disentangled Curriculum for Strategic Perception in Multimodal Reasoning
Siqi Yang, Zilve Gao, Haibo Qiu +5
Multimodal Large Language Models (MLLMs) demonstrate significant potential but remain brittle in complex, long-chain visual reasoning tasks. A critical failure mode is "visual forg…
Reading or Reasoning? Format Decoupled Reinforcement Learning for Document OCR
Yufeng Zhong, Lei Chen, Zhixiong Zeng +8
Reading text from images or scanned documents via OCR models has been a longstanding focus of researchers. Intuitively, text reading is perceived as a straightforward perceptual ta…
Context Cascade Compression: Exploring the Upper Limits of Text Compression
Fanfan Liu, Haibo Qiu
Million-level token inputs in long-context tasks pose significant computational and memory challenges for Large Language Models (LLMs). Recently, DeepSeek-OCR conducted research in…
Metis-HOME: Hybrid Optimized Mixture-of-Experts for Multimodal Reasoning
Xiaohan Lan, Fanfan Liu, Haibo Qiu +4
Inspired by recent advancements in LLM reasoning, the field of multimodal reasoning has seen remarkable progress, achieving significant performance gains on intricate tasks such as…
Perceptual-Evidence Anchored Reinforced Learning for Multimodal Reasoning
Chi Zhang, Haibo Qiu, Qiming Zhang +6
Reinforcement Learning with Verifiable Rewards (RLVR) has significantly advanced the reasoning capabilities of Large Language Models (LLMs) and is now being applied to Vision-Langu…
VinciCoder: Unifying Multimodal Code Generation via Coarse-to-fine Visual Reinforcement Learning
Xuanle Zhao, Deyang Jiang, Zhixiong Zeng +8
While recent specialized multimodal code generation models excel in tasks like chart-to-code generation, their reliance on single-task training limits generalization and hinders th…
Towards Better & Faster Autoregressive Image Generation: From the Perspective of Entropy
Xiaoxiao Ma, Feng Zhao, Pengyang Ling +6
In this work, we first revisit the sampling issues in current autoregressive (AR) image generation models and identify that image tokens, unlike text tokens, exhibit lower informat…
Metis-SPECS: Decoupling Multimodal Learning via Self-distilled Preference-based Cold Start
Kun Chen, Peng Shi, Haibo Qiu +4
Reinforcement learning (RL) with verifiable rewards has recently catalyzed a wave of "MLLM-r1" approaches that bring RL to vision language models. Most representative paradigms beg…
STAGE: Stable and Generalizable GRPO for Autoregressive Image Generation
Xiaoxiao Ma, Haibo Qiu, Guohui Zhang +4
Reinforcement learning has recently been explored to improve text-to-image generation, yet applying existing GRPO algorithms to autoregressive (AR) image models remains challenging…
OmniActor: A Generalist GUI and Embodied Agent for 2D&3D Worlds
Longrong Yang, Zhixiong Zeng, Yufeng Zhong +7
Multimodal large language models are evolving toward multimodal agents capable of proactively executing tasks. Most agent research focuses on GUI or embodied scenarios, which corre…
DeepSketcher: Internalizing Visual Manipulation for Multimodal Reasoning
Chi Zhang, Haibo Qiu, Qiming Zhang +3
The "thinking with images" paradigm represents a pivotal shift in the reasoning of Vision Language Models (VLMs), moving from text-dominant chain-of-thought to image-interactive re…
Metis-RISE: RL Incentivizes and SFT Enhances Multimodal Reasoning Model Learning
Haibo Qiu, Xiaohan Lan, Fanfan Liu +4
Recent advancements in large language models (LLMs) have witnessed a surge in the development of advanced reasoning paradigms, which are now being integrated into multimodal large…
UITron-Speech: Towards Automated GUI Agents Based on Speech Instructions
Wenkang Han, Zhixiong Zeng, Jing Huang +7
Autonomous agents for Graphical User Interfaces (GUIs) are revolutionizing human-computer interaction, yet their reliance on text-based instructions imposes limitations on accessib…
UniToken: Harmonizing Multimodal Understanding and Generation through Unified Visual Encoding
Yang Jiao, Haibo Qiu, Zequn Jie +4
We introduce UniToken, an auto-regressive generation model that encodes visual inputs through a combination of discrete and continuous representations, enabling seamless integratio…