10 papers
Asking What a Director Sees Can Change What Vision-Language Models Choose
Jiaxuan Guo, Xingyu Ren, Kejia Zhang +3
Vision-language models are now routinely tested on theory of mind: whether they can work out what another person sees, knows, and means. A common design asks the model to report th…
Scivot: Benchmarking Principal-Investigator Judgment at Pivotal Moments of Scientific AutoResearch Agents
Youran Sun, Xingyu Ren, Chugang Yi +15
Large language model (LLM) agents can keep scientific experiments running for months with little human supervision, but reliable autonomous research also requires scientific judgme…
When an Evaluation Rule Writes Training Labels: Measuring Human-Reference Forgiveness in NAVSIM
Jiaxuan Guo, Jingxin Yang, Jiaqi Ye +4
When the human reference scores zero on a metric, the released GTRS-Dense label generator for NAVSIM marks every candidate trajectory in the scene as passing it. NAVSIM's authors i…
Scoring the Wrong Question: Readout Failures in Constrained-Option Evaluation
Jiaxuan Guo, Kejia Zhang, Shuo Xin +3
Constrained-option scoring reads a model's probabilities for a fixed set of permitted answers, so it returns a score even when the model is about to write something else. We study…
AutoSR: Automatic Symbolic Regression by Searching Research States
Kejia Zhang, Youran Sun, Xinyu Ren +2
We introduce Automatic Symbolic Regression (AutoSR), a fully automated system that instantiates Research-Space Symbolic Regression by searching persistent scientific investigations…
Cross-Task Dissociation in Frontier Vision-Language Model Theory of Mind
Kejia Zhang, Youran Sun, Chugang Yi +1
Do frontier vision-language models present a coherent Theory-of-Mind (ToM) profile across tasks, matching the same human reference group, or does that profile fragment from one par…
Reasoning or Memorization: Can LLMs Understand and Generate Chinese Xiehouyu Riddles?
Hai Hu, Siyuan Song, Chongtian Shao +3
In this paper, we push the boundary of LLM reasoning by testing them in a Chinese language game, xiehouyu, with novel xiehouyu created by linguists that had not existed before to a…
PerspectiveGap: A Benchmark for Multi-Agent Orchestration Prompting
Youran Sun, Xingyu Ren, Kejia Zhang +2
Real-world LLM applications are moving beyond single-agent workflows toward orchestrated multi-agent systems, yet current models still struggle to determine what each sub-agent nee…
Agon: An Autonomous Large-Scale Omnidisciplinary Research System Built on Prompt Economy
Youran Sun, Xingyu Ren, Chugang Yi +4
Large language models are making research production scalable, shifting the bottleneck from producing artifacts to judging claims. We present \textsc{Agon}, a research orchestrator…
A Systematic Assessment of Language Models with Linguistic Minimal Pairs in Chinese
Yikang Liu, Yeting Shen, Hongao Zhu +9
We present ZhoBLiMP, the largest linguistic minimal pair benchmark for Chinese, with over 100 paradigms, ranging from topicalization to the \textit{Ba} construction. We then train…