7 papers
Overview of the NLPCC 2026 Shared Task 11: Agent-Based Experiment Reproduction from Scientific Papers
Hanhua Hong, Yizhi Li, Luu Gia Huy +3
Reproducibility is essential to scientific progress, yet the growing volume and complexity of scientific publications make exhaustive manual verification increasingly impractical.…
Can LLMs Write Reliable Rubrics? A Meta-Evaluation for Experiment Reproduction
Hanhua Hong, Yizhi Li, Jiaoyan Chen +4
Rubric-based evaluation is a promising approach for assessing open-ended outputs from LLM-based research agents, particularly in paper reproduction, where direct paper-to-repositor…
HiRAS: A Hierarchical Multi-Agent Framework for Paper-to-Code Generation and Execution
Hanhua Hong, Yizhi LI, Jiaoyan Chen +4
Recent advances in large language models have highlighted their potential to automate computational research, particularly reproducing experimental results. However, existing appro…
Overview of the NLPCC 2025 Shared Task: Gender Bias Mitigation Challenge
Yizhi Li, Ge Zhang, Hanhua Hong +2
As natural language processing for gender bias becomes a significant interdisciplinary topic, the prevalent data-driven techniques, such as pre-trained language models, suffer from…
Beyond One-Size-Fits-All: Inversion Learning for Highly Effective NLG Evaluation Prompts
Hanhua Hong, Chenghao Xiao, Yang Wang +3
Evaluating natural language generation systems is challenging due to the diversity of valid outputs. While human evaluation is the gold standard, it suffers from inconsistencies, l…
Leveraging Estimated Transferability Over Human Intuition for Model Selection in Text Ranking
Jun Bai, Zhuofan Chen, Zhenzi Li +5
Text ranking has witnessed significant advancements, attributed to the utilization of dual-encoder enhanced by Pre-trained Language Models (PLMs). Given the proliferation of availa…
How to Determine the Most Powerful Pre-trained Language Model without Brute Force Fine-tuning? An Empirical Survey
Jun Bai, Xiaofeng Zhang, Chen Li +4
Transferability estimation has been attached to great attention in the computer vision fields. Researchers try to estimate with low computational cost the performance of a model wh…