17 papers
AgenticSTS: A Bounded-Memory Testbed for Long-Horizon LLM Agents
Xiangchen Cheng, Yunwei Jiang, Jianwen Sun +7
Memory for a long-horizon LLM agent is a contract about what each future decision is allowed to see. The simplest contract appends past observations, tool calls, and reflections to…
JAMER: Project-Level Code Framework Dataset and Benchmark on Professional Game Engines
Jianwen Sun, Chuanhao Li, Zizhen Li +5
Current AI-driven game development has made substantial progress in asset generation, gameplay design, and web-based game coding, yet project-level code engineering on professional…
CoBG: Learning from Player Reviews to Design, Critique, and Revise Board Game Rulebooks
Zizhen Li, Chuanhao Li, Yibin Wang +6
Board game design alternates between thinking as a designer and experiencing as a player, over repeated cycles of prototyping, critique, and revision. This loop is cognitively dema…
LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces
Yukang Feng, Jianwen Sun, Zelai Yang +16
Recent advances in AI-assisted programming have empowered agents to execute complex workflows via command-line interfaces, however, existing benchmarks are limited by short task ho…
World Craft: Agentic Framework to Create Visualizable Worlds via Text
Jianwen Sun, Yukang Feng, Kaining Ying +8
Large Language Models (LLMs) motivate generative agent simulation (e.g., AI Town) to create a ``dynamic world'', holding immense value across entertainment and research. However, f…
MeepleLM: A Virtual Playtester Simulating Diverse Subjective Experiences
Zizhen Li, Chuanhao Li, Yibin Wang +7
Recent advancements have expanded the role of Large Language Models in board games from playing agents to creative co-designers. However, a critical gap remains: current systems la…
ProSoftArena: Benchmarking Hierarchical Capabilities of Multimodal Agents in Professional Software Environments
Jiaxin Ai, Yukang Feng, Fanrui Zhang +8
Multimodal agents are making rapid progress on general computer-use tasks, yet existing benchmarks remain largely confined to browsers and basic desktop applications, falling short…
Code-in-the-Loop Forensics: Agentic Tool Use for Image Forgery Detection
Fanrui Zhang, Qiang Zhang, Sizhuo Zhou +10
Existing image forgery detection (IFD) methods either exploit low-level, semantics-agnostic artifacts or rely on multimodal large language models (MLLMs) with high-level semantic k…
From Pixels to Paths: A Multi-Agent Framework for Editable Scientific Illustration
Jianwen Sun, Fanrui Zhang, Yukang Feng +6
Scientific illustrations demand both high information density and post-editability. However, current generative models have two major limitations: Frist, image generation models ou…
Closing the Expression Gap in LLM Instructions via Socratic Questioning
Jianwen Sun, Yukang Feng, Yifan Chang +6
A fundamental bottleneck in human-AI collaboration is the ``intention expression gap," the difficulty for humans to effectively convey complex, high-dimensional thoughts to AI. Thi…
InMind: Evaluating LLMs in Capturing and Applying Individual Human Reasoning Styles
Zizhen Li, Chuanhao Li, Yibin Wang +8
LLMs have shown strong performance on human-centric reasoning tasks. While previous evaluations have explored whether LLMs can infer intentions or detect deception, they often over…
MDK12-Bench: A Comprehensive Evaluation of Multimodal Large Language Models on Multidisciplinary Exams
Pengfei Zhou, Xiaopeng Peng, Fanrui Zhang +18
Multimodal large language models (MLLMs), which integrate language and visual cues for problem-solving, are crucial for advancing artificial general intelligence (AGI). However, cu…
Sekai: A Video Dataset towards World Exploration
Zhen Li, Chuanhao Li, Xiaofeng Mao +17
Video generation techniques have made remarkable progress, promising to be the foundation of interactive world exploration. However, existing video generation datasets are not well…
A High-Quality Dataset and Reliable Evaluation for Interleaved Image-Text Generation
Yukang Feng, Jianwen Sun, Chuanhao Li +8
Recent advancements in Large Multimodal Models (LMMs) have significantly improved multimodal understanding and generation. However, these models still struggle to generate tightly…
MDK12-Bench: A Multi-Discipline Benchmark for Evaluating Reasoning in Multimodal Large Language Models
Pengfei Zhou, Fanrui Zhang, Xiaopeng Peng +17
Multimodal reasoning, which integrates language and visual cues into problem solving and decision making, is a fundamental aspect of human intelligence and a crucial step toward ar…
ProJudge: A Multi-Modal Multi-Discipline Benchmark and Instruction-Tuning Dataset for MLLM-based Process Judges
Jiaxin Ai, Pengfei Zhou, Zhaopan Xu +8
As multi-modal large language models (MLLMs) frequently exhibit errors when solving scientific problems, evaluating the validity of their reasoning processes is critical for ensuri…
ARMOR: Empowering Multimodal Understanding Model with Interleaved Multimodal Generation Capability
Jianwen Sun, Yukang Feng, Chuanhao Li +7
Unified multimodal understanding and generation have recently received much attention in the area of vision and language. Existing UniMs are designed to simultaneously learn both m…