5 papers
A Unified Evaluation Framework for Multi-Annotator Tendency Learning
Liyun Zhang, Fengkai Liu, Xuanmeng Sha +3
Recent works have emerged in multi-annotator learning that shift focus from Consensus-oriented Learning (CoL), which aggregates multiple annotations into a single ground-truth pred…
QuMAB: Query-based Multi-Annotator Behavior Modeling with Reliability under Sparse Labels
Liyun Zhang, Zheng Lian, Hong Liu +2
Multi-annotator learning traditionally aggregates diverse annotations to approximate a single ground truth, treating disagreements as noise. However, this paradigm faces fundamenta…
SimLabel: Similarity-Weighted Iterative Framework for Multi-annotator Learning with Missing Annotations
Liyun Zhang, Zheng Lian, Hong Liu +2
Multi-annotator learning (MAL) aims to model annotator-specific labeling patterns. However, existing methods face a critical challenge: they simply skip updating annotator-specific…
REPRO-Bench: Can Agentic AI Systems Assess the Reproducibility of Social Science Research?
Chuxuan Hu, Liyun Zhang, Yeji Lim +3
Assessing the reproducibility of social science papers is essential for promoting rigor in research processes, but manual assessment is costly. With recent advances in agentic AI s…
QuMATL: Query-based Multi-annotator Tendency Learning
Liyun Zhang, Zheng Lian, Hong Liu +2
Different annotators often assign different labels to the same sample due to backgrounds or preferences, and such labeling patterns are referred to as tendency. In multi-annotator…