5 papers
CrowdAgent: Multi-Agent Managed Multi-Source Annotation System
Maosheng Qin, Renyu Zhu, Mingxuan Xia +8
High-quality annotated data is a cornerstone of modern Natural Language Processing (NLP). While recent methods begin to leverage diverse annotation sources-including Large Language…
The Effects of Data Augmentation on Confidence Estimation for LLMs
Rui Wang, Renyu Zhu, Minmin Lin +4
Confidence estimation is crucial for reflecting the reliability of large language models (LLMs), particularly in the widely used closed-source models. Utilizing data augmentation f…
Digital Player: Evaluating Large Language Models based Human-like Agent in Games
Jiawei Wang, Kai Wang, Shaojie Lin +11
With the rapid advancement of Large Language Models (LLMs), LLM-based autonomous agents have shown the potential to function as digital employees, such as digital analysts, teacher…
A Dataset for the Validation of Truth Inference Algorithms Suitable for Online Deployment
Fei Wang, Haoyu Liu, Haoyang Bi +9
For the purpose of efficient and cost-effective large-scale data labeling, crowdsourcing is increasingly being utilized. To guarantee the quality of data labeling, multiple annotat…
Towards Long-term Annotators: A Supervised Label Aggregation Baseline
Haoyu Liu, Fei Wang, Minmin Lin +6
Relying on crowdsourced workers, data crowdsourcing platforms are able to efficiently provide vast amounts of labeled data. Due to the variability in the annotation quality of crow…