11 papers
Beyond Simply Environment Scaling: Designing Effective Environment Distributions for Multimodal Agent Learning
Kejian Zhu, Zhuoran Jin, Dongqi Huang +4
Recent works train agents by constructing large-scale multimodal environment pools. However, we find that simply increasing the number of multimodal environments does not always be…
Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix It
Yupu Hao, Zhuoran Jin, Huanxuan Liao +2
Tool use enables large language models (LLMs) to perform complex tasks, and recent agentic reinforcement learning (RL) methods show promise for enhancing model capabilities. Howeve…
Look Light, Think Heavy: What Multimodal Chain-of-Thought Reasoning Can and Cannot Do
Zhuoran Jin, Kejian Zhu, Hongbang Yuan +5
Chain-of-Thought (CoT) has become a standard method for improving reasoning capabilities in large language models (LLMs) by eliciting step-by-step thinking, but its effectiveness i…
Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Application
Jiachun Li, Zhuoran Jin, Tianyi Men +12
Environments serve as interactive systems for large language model (LLM) based agents across diverse scenarios and play a crucial role in driving the continual evolution of model c…
Pushing the Limits of LLM Tool Calling via Experiential Knowledge Integration and Activation
Yupu Hao, Zhuoran Jin, Huanxuan Liao +2
Large language models (LLMs) rely on tool use to act as autonomous agents, yet often fail in multi-step execution due to insufficient tool-related knowledge and ineffective knowled…
ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning
Huanxuan Liao, Zhongtao Jiang, Yupu Hao +6
Multimodal Large Language Models (MLLMs) achieve stronger visual understanding by scaling input fidelity, yet the resulting visual token growth makes jointly sustaining high spatia…
Think While Watching: Online Streaming Segment-Level Memory for Multi-Turn Video Reasoning in Multimodal Large Language Models
Lu Wang, Zhuoran Jin, Yupu Hao +4
Multimodal large language models (MLLMs) have shown strong performance on offline video understanding, but most are limited to offline inference or have weak online reasoning, maki…
Evaluating Personalized Tool-Augmented LLMs from the Perspectives of Personalization and Proactivity
Yupu Hao, Pengfei Cao, Zhuoran Jin +4
Personalized tool utilization is essential for aligning large language models (LLMs) with user preference in interaction scenarios with various tools. However, most of the current…
DATA: Decomposed Attention-based Task Adaptation for Rehearsal-Free Continual Learning
Huanxuan Liao, Shizhu He, Yupu Hao +2
Continual learning (CL) is essential for Large Language Models (LLMs) to adapt to evolving real-world demands, yet they are susceptible to catastrophic forgetting (CF). While tradi…
CITI: Enhancing Tool Utilizing Ability in Large Language Models without Sacrificing General Performance
Yupu Hao, Pengfei Cao, Zhuoran Jin +4
Tool learning enables the Large Language Models (LLMs) to interact with the external environment by invoking tools, enriching the accuracy and capability scope of LLMs. However, pr…
: Internalizing Symbolic Knowledge for Distilling Better CoT Capabilities into Small Language Models
Huanxuan Liao, Shizhu He, Yupu Hao +4
Small Language Models (SLMs) are attracting attention due to the high computational demands and privacy concerns of Large Language Models (LLMs). Some studies fine-tune SLMs using…