7 papers
Towards On-Policy Data Evolution for Visual-Native Multimodal Deep Search Agents
Shijue Huang, Hangyu Guo, Guanting Dong +8
Multimodal deep search requires an agent to solve open-world problems by chaining search, tool use, and visual reasoning over evolving textual and visual context. Two bottlenecks l…
Unsupervised Process Reward Models
Artyom Gadetsky, Maxim Kodryan, Siba Smarak Panigrahi +2
Process Reward Models (PRMs) are a powerful mechanism for steering large language model reasoning by providing fine-grained, step-level supervision. However, this effectiveness com…
RDKV: Rate-Distortion Bit Allocation for Joint Eviction and Quantization of the KV Cache
Junkai Zhang, Hang Guo, Luca Benini +1
Large language models (LLMs) have shown strong performance across diverse tasks, but their inference with long input contexts is bottlenecked by memory size and bandwidth. The Key-…
AgentVista: Evaluating Multimodal Agents in Ultra-Challenging Realistic Visual Scenarios
Zhaochen Su, Jincheng Gao, Hangyu Guo +10
Real-world multimodal agents solve multi-step workflows grounded in visual evidence. For example, an agent can troubleshoot a device by linking a wiring photo to a schematic and va…
Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers
Zhaochen Su, Peng Xia, Hangyu Guo +12
Recent progress in multimodal reasoning has been significantly advanced by textual Chain-of-Thought (CoT), a paradigm where models conduct reasoning within language. This text-cent…
What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuning
Yifan Du, Hangyu Guo, Kun Zhou +6
Visual instruction tuning is crucial for enhancing the zero-shot generalization capability of Multi-modal Large Language Models (MLLMs). In this paper, we aim to investigate a fund…