6 papers
CrowdAgent: Multi-Agent Managed Multi-Source Annotation System
Maosheng Qin, Renyu Zhu, Mingxuan Xia +8
High-quality annotated data is a cornerstone of modern Natural Language Processing (NLP). While recent methods begin to leverage diverse annotation sources-including Large Language…
The Effects of Data Augmentation on Confidence Estimation for LLMs
Rui Wang, Renyu Zhu, Minmin Lin +4
Confidence estimation is crucial for reflecting the reliability of large language models (LLMs), particularly in the widely used closed-source models. Utilizing data augmentation f…
Rank Aggregation in Crowdsourcing for Listwise Annotations
Wenshui Luo, Haoyu Liu, Yongliang Ding +7
Rank aggregation through crowdsourcing has recently gained significant attention, particularly in the context of listwise ranking annotations. However, existing methods primarily f…
A Dataset for the Validation of Truth Inference Algorithms Suitable for Online Deployment
Fei Wang, Haoyu Liu, Haoyang Bi +9
For the purpose of efficient and cost-effective large-scale data labeling, crowdsourcing is increasingly being utilized. To guarantee the quality of data labeling, multiple annotat…
FreeAL: Towards Human-Free Active Learning in the Era of Large Language Models
Ruixuan Xiao, Yiwen Dong, Junbo Zhao +4
Collecting high-quality labeled data for model training is notoriously time-consuming and labor-intensive for various NLP tasks. While copious solutions, such as active learning fo…
Towards Long-term Annotators: A Supervised Label Aggregation Baseline
Haoyu Liu, Fei Wang, Minmin Lin +6
Relying on crowdsourced workers, data crowdsourcing platforms are able to efficiently provide vast amounts of labeled data. Due to the variability in the annotation quality of crow…