Measuring the Effect of Disfluency in Multilingual Knowledge Probing Benchmarks

Kirill Semenov, Rico Sennrich

For multilingual factual knowledge assessment of LLMs, benchmarks such as MLAMA use template translations that do not take into account the grammatical and semantic information of…