28 citations · 128 across the 48 of their papers we have counts for
24 papers · 1 filter
Can LLMs Write Reliable Rubrics? A Meta-Evaluation for Experiment Reproduction
Hanhua Hong, Yizhi Li, Jiaoyan Chen +4
Rubric-based evaluation is a promising approach for assessing open-ended outputs from LLM-based research agents, particularly in paper reproduction, where direct paper-to-repositor…
Encyclo-K: Evaluating LLMs with Dynamically Composed Knowledge Statements
Yiming Liang, Yizhi Li, Yantao Du +14
Benchmarks play a crucial role in tracking the rapid advancement of large language models (LLMs) and identifying their capability boundaries. However, existing benchmarks predomina…
CodeSimpleQA: Scaling Factuality in Code Large Language Models
Jian Yang, Wei Zhang, Yizhi Li +8
Large language models (LLMs) have made significant strides in code generation, achieving impressive capabilities in synthesizing code snippets from natural language instructions. H…
Adversarial Defence without Adversarial Defence: Enhancing Language Model Robustness via Instance-level Principal Component Removal
Yang Wang, Chenghao Xiao, Yizhi Li +3
Pre-trained language models (PLMs) have driven substantial progress in natural language processing but remain vulnerable to adversarial attacks, raising concerns about their robust…
Re:Form -- Reducing Human Annotations in Scalable Formal Software Verification with RL in LLMs: A Preliminary Study on Dafny
Chuanhao Yan, Fengdi Che, Xuhan Huang +12
Existing informal language-based (e.g., human language) Large Language Models (LLMs) trained with Reinforcement Learning (RL) face a significant challenge: their verification proce…
Overview of the NLPCC 2025 Shared Task: Gender Bias Mitigation Challenge
Yizhi Li, Ge Zhang, Hanhua Hong +2
As natural language processing for gender bias becomes a significant interdisciplinary topic, the prevalent data-driven techniques, such as pre-trained language models, suffer from…
SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines
P Team, Xinrun Du, Yifan Yao +94
Large language models (LLMs) have demonstrated remarkable proficiency in mainstream academic disciplines such as mathematics, physics, and computer science. However, human knowledg…
LongEval: A Comprehensive Analysis of Long-Text Generation Through a Plan-based Paradigm
Siwei Wu, Yizhi Li, Xingwei Qu +7
Large Language Models (LLMs) have achieved remarkable success in various natural language processing tasks, yet their ability to generate long-form content remains poorly understoo…
MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale
Jarvis Guo, Tuney Zheng, Yuelin Bai +7
Open-source multimodal large language models (MLLMs) have shown significant potential in a broad range of multimodal tasks. However, their reasoning capabilities remain constrained…
A Comparative Study on Reasoning Patterns of OpenAI's o1 Model
Siwei Wu, Zhongyuan Peng, Xinrun Du +14
Enabling Large Language Models (LLMs) to handle a wider range of complex tasks (e.g., coding, math) has drawn great attention from many researchers. As LLMs continue to evolve, mer…
MIO: A Foundation Model on Multimodal Tokens
Zekun Wang, King Zhu, Chunpu Xu +14
In this paper, we introduce MIO, a novel foundation model built on multimodal tokens, capable of understanding and generating speech, text, images, and videos in an end-to-end, aut…
OmniBench: Towards The Future of Universal Omni-Language Models
Yizhi Li, Yinghao Ma, Ge Zhang +20
Recent advancements in multimodal large language models (MLLMs) have aimed to integrate and interpret data across diverse modalities. However, the capacity of these models to concu…
Overview of the NLPCC 2024 Shared Task on Chinese Metaphor Generation
Xingwei Qu, Ge Zhang, Siwei Wu +2
This paper presents the results of the shared task on Chinese metaphor generation, hosted at the 13th CCF Conference on Natural Language Processing and Chinese Computing (NLPCC 202…
MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Series
Ge Zhang, Scott Qu, Jiaheng Liu +42
Large Language Models (LLMs) have made great strides in recent years to achieve unprecedented performance across different tasks. However, due to commercial interest, the most comp…
The Fine Line: Navigating Large Language Model Pretraining with Down-streaming Capability Analysis
Chen Yang, Junzhuo Li, Xinyao Niu +11
Uncovering early-stage metrics that reflect final model performance is one core principle for large-scale pretraining. The existing scaling law demonstrates the power-law correlati…
Pixel Sentence Representation Learning
Chenghao Xiao, Zhuoxu Huang, Danlu Chen +7
Pretrained language models are long known to be subpar in capturing sentence and document-level semantics. Though heavily investigated, transferring perturbation-based methods from…
CIF-Bench: A Chinese Instruction-Following Benchmark for Evaluating the Generalizability of Large Language Models
Yizhi LI, Ge Zhang, Xingwei Qu +16
The advancement of large language models (LLMs) has enhanced the ability to generalize across a wide range of unseen natural language processing (NLP) tasks through instruction-fol…
CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark
Ge Zhang, Xinrun Du, Bei Chen +19
As the capabilities of large multimodal models (LMMs) continue to advance, evaluating the performance of LMMs emerges as an increasing need. Additionally, there is an even larger g…
Length is a Curse and a Blessing for Document-level Semantics
Chenghao Xiao, Yizhi Li, G Thomas Hudson +2
In recent years, contrastive learning (CL) has been extensively utilized to recover sentence and document-level encoding capability from pre-trained language models. In this work,…
LyricWhiz: Robust Multilingual Zero-shot Lyrics Transcription by Whispering to ChatGPT
Le Zhuo, Ruibin Yuan, Jiahao Pan +10
We introduce LyricWhiz, a robust, multilingual, and zero-shot automatic lyrics transcription method achieving state-of-the-art performance on various lyrics transcription datasets,…
Interactive Natural Language Processing
Zekun Wang, Ge Zhang, Kexin Yang +19
Interactive Natural Language Processing (iNLP) has emerged as a novel paradigm within the field of NLP, aimed at addressing limitations in existing frameworks while aligning with t…
Chinese Open Instruction Generalist: A Preliminary Release
Ge Zhang, Yemin Shi, Ruibo Liu +9
Instruction tuning is widely recognized as a key technique for building generalist language models, which has attracted the attention of researchers and the public with the release…
CORGI-PM: A Chinese Corpus For Gender Bias Probing and Mitigation
Ge Zhang, Yizhi Li, Yaoyao Wu +5
As natural language processing (NLP) for gender bias becomes a significant interdisciplinary topic, the prevalent data-driven techniques such as large-scale language models suffer…
HERB: Measuring Hierarchical Regional Bias in Pre-trained Language Models
Yizhi Li, Ge Zhang, Bohao Yang +4
Fairness has become a trending topic in natural language processing (NLP), which addresses biases targeting certain social groups such as genders and religions. However, regional b…