4 citations · 5 across the 15 of their papers we have counts for
21 papers
TRIAD: Benchmarking Omni-Modal Ambiguity in Multimodal Large Language Models
Zhaolu Kang, Yidi Wang, Yile Li +4
Ambiguity is a central feature of natural communication: a sentence, image, or sound may remain underspecified until evidence from the other channels is considered. Yet current mul…
DRILL: Training World Models to Improve Policies, Not Predict Pixels
Zhaolu Kang, Tailong Luo, Lei Wei +9
World models are increasingly used to reduce environment interaction when post-training vision-language-action (VLA) policies with reinforcement learning. Yet they are typically tr…
AgentPerturb: Evaluating LLM Agent Robustness under Controlled Interaction Perturbations
Yuanjian Xu, Jinyi Han, Jianing Hao +5
Real-world tasks are often complex and noisy, posing substantial challenges for LLM agents. Yet existing evaluations lack a systematic framework for jointly characterizing potentia…
Rethinking the Role of Tool Graph Memory in Multi-Turn Tool Use
Uiji Hwang, Richeng Xuan, Taeuk Kim
Large language model (LLM) agents solve complex tasks by calling external tools over multiple turns. Recent work has proposed tool graph memory, which stores past experience as too…
JPO: Juris Policy Optimization for Structured Legal Reasoning in Criminal Judgment Prediction
Zhaolu Kang, Yantao Liu, Tailong Luo +14
Criminal judgment prediction requires models to infer statutory articles, charges, and sentencing outcomes from case facts. Unlike standard classification tasks, it involves a stru…
Modality Fault Lines: Structural Corruptions Reveal Fragile Omni-Modal Reasoning
Zhaolu Kang, Meixin Wu, Yu Xue +6
Omni-modal large language models are increasingly evaluated on clean text--vision--audio inputs, where every channel is present, synchronized, and readily interpretable. Such score…
TRAJDEBUG: Tracing Error Lifecycle to Identify Critical Failures in Long-Horizon Agent Trajectories
Yunjia Qi, Zehua Yin, Xintong Shi +10
LLM-based agentic systems have shown remarkable capabilities in complex domains, while suffering from cascading errors and difficulty in debugging. Critical error detection aims to…
RepoProbe: Benchmarking Architecture-Aware Repository Comprehension with Checklists
Yuexi Yang, Alyssa Wu, Ji Luo +4
The integration of Large Language Models (LLMs) into software engineering has shifted the focus from function-level generation to repository-scale assistance. However, existing ben…
Can LLM-as-a-Judge Reliably Verify Rubrics in Agentic Scenarios?
Yangda Peng, Yunjia Qi, Haotian Xia +8
Rubric-based scoring has become a widely used paradigm in model evaluation, typically with LLM-as-a-Judge (LaaJ) for rubric scoring. However, the reliability of LaaJ for rubric sco…
VisualNeedle: Benchmarking Active Visual Search in Information-Dense Scenes
Jingru Chen, Yiming Liu, Mingtao Chen +5
Frontier multimodal large language models (MLLMs) have been reported to achieve over 90\% accuracy on fine-grained perception benchmarks. However, such scores do not necessarily im…
Trust the uncertain teacher: distilling dark knowledge via calibrated uncertainty
Jeonghyun Kim, SooKyung Kim, Richeng Xuan +1
The core of knowledge distillation lies in transferring the teacher's rich 'dark knowledge'-subtle probabilistic patterns that reveal how classes are related and the distribution o…
QuantEval: A Benchmark for Financial Quantitative Tasks in Large Language Models
Zhaolu Kang, Junhao Gong, Wenqing Hu +15
Large Language Models (LLMs) have shown strong capabilities in finance-oriented question answering, yet their evaluation on professional quantitative workflows remains fragmented a…
LaoBench: A Large-Scale Multidimensional Lao Benchmark for Large Language Models
Jian Gao, Richeng Xuan, Zhaolu Kang +9
The rapid advancement of large language models (LLMs) has not been matched by their evaluation in low-resource languages, especially Southeast Asian languages like Lao. To fill thi…
FlagEval Findings Report: A Preliminary Evaluation of Large Reasoning Models on Automatically Verifiable Textual and Visual Questions
Bowen Qin, Chen Yue, Fang Yin +26
We conduct a moderate-scale contamination-free (to some extent) evaluation of current large reasoning models (LRMs) with some preliminary findings. We also release ROME, our evalua…
Overconfidence in LLM-as-a-Judge: Diagnosis and Confidence-Driven Solution
Zailong Tian, Zhuoheng Han, Yanzhe Chen +5
Large Language Models (LLMs) are widely used as automated judges, where practical value depends on both accuracy and trustworthy, risk-aware judgments. Existing approaches predomin…
OMHBench: Benchmarking Balanced and Grounded Omni-Modal Multi-Hop Reasoning
Seunghee Kim, Ingyu Bang, Seokgyu Jang +5
Multimodal Large Language Models (MLLMs) have increasingly supported omni-modal processing across text, vision, and speech. However, existing evaluation frameworks for such models…
FlagEvalMM: A Flexible Framework for Comprehensive Multimodal Model Evaluation
Zheqi He, Yesheng Liu, Jing-shu Zheng +5
We present FlagEvalMM, an open-source evaluation framework designed to comprehensively assess multimodal models across a diverse range of vision-language understanding and generati…
Memorization or Reasoning? Exploring the Idiom Understanding of LLMs
Jisu Kim, Youngwoo Shin, Uiji Hwang +3
Idioms have long posed a challenge due to their unique linguistic properties, which set them apart from other common expressions. While recent studies have leveraged large language…
HalluDial: A Large-Scale Benchmark for Automatic Dialogue-Level Hallucination Evaluation
Wen Luo, Tianshu Shen, Wei Li +4
Large Language Models (LLMs) have significantly advanced the field of Natural Language Processing (NLP), achieving remarkable performance across diverse tasks and enabling widespre…
Bi-DCSpell: A Bi-directional Detector-Corrector Interactive Framework for Chinese Spelling Check
Haiming Wu, Hanqing Zhang, Richeng Xuan +1
Chinese Spelling Check (CSC) aims to detect and correct potentially misspelled characters in Chinese sentences. Naturally, it involves the detection and correction subtasks, which…
CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning
Zheqi He, Xinya Wu, Pengfei Zhou +5
Multi-modal large language models(MLLMs) have achieved remarkable progress and demonstrated powerful knowledge comprehension and reasoning abilities. However, the mastery of domain…