Showing cs.CLShow all
3 papers · 1 filter
cs.CL2024
VariErr NLI: Separating Annotation Error from Human Label Variation
Leon Weber-Genzel, Siyao Peng, Marie-Catherine de Marneffe +1
Human label variation arises when annotators assign different labels to the same item for valid reasons, while annotation errors occur when labels are assigned for invalid reasons.…
cs.CL2024
"My Answer is C": First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Models
Xinpeng Wang, Bolei Ma, Chengzhi Hu +5
The open-ended nature of language generation makes the evaluation of autoregressive large language models (LLMs) challenging. One common evaluation approach uses multiple-choice qu…
cs.CL2023
Donkii: Can Annotation Error Detection Methods Find Errors in Instruction-Tuning Datasets?
Leon Weber-Genzel, Robert Litschko, Ekaterina Artemova +1
Instruction tuning has become an integral part of training pipelines for Large Language Models (LLMs) and has been shown to yield strong performance gains. In an orthogonal line of…